CHALLENGE DETAIL

Salesforce 2 Salesforce "Data Dumper"

OVERVIEW

CODE

One of our frequent tasks at CloudSpokes is to spin up Force.com Developer orgs for challenge participants to use during a specific challenge. We typically use ANT to install code and configurations but importing data is always a chore. We want to build a tool that runs on a local machine or in the cloud that takes data from a source salesforce.com org and dumps it into target salesforce org.

The tool will allow you to specify multiple objects, the fields to query for each object and the credentials for both the source and target orgs. The records would always be upserted in the target org. We also want to have the ability to obfuscate specific field data. For instance, when dumping data from production to a developer org, we typically change email addresses to something like “noemail@cloudspokes.com”.

We currently use roughly the same process to move data from SQL Server to our database.com org on a nightly basis. The code is in Java and you can find it on this GitHub repo. It should make a great starter app.

REQUIREMENTS

Use the Force.com Web Services API or Bulk API to transfer data (see Additional Info section for relevant docs). We this this is a great job for the Bulk API.

Use any language that you’d like to build this tool however there is a lot of sample code in Java. The tool does not need to have any sort of UI; we’d prefer to run it from the command line. The tool will read a config/xml file which contains all data for a particular job including but not limited to:

- source connection info
- target connection info
- object/table with data -- must support multiple objects
- fields (or query)
- where clause
- external id
- n number of fields to be obfuscated and the value (email__c => ‘noemail@cloudspokes.com’, phone => ‘111-222-3333’)

When the process runs it will read the config/xml file, query the source org for data and push the data into the target org. You will need to allow data to be upserted using externalIds. Assume that both the source and target orgs have the same data structure for the objects. All records will always be upserted into the target org.

Submission
Upload all your source code as a zip and provide any documentation and/or instructions that are needed. Please be clear and concise with any setup instructions.

An SWF video of your application using Jing would also be helpful but not required.

SUBMISSION DETAILS

Upload all your source code as a zip and provide any documentation and/or instructions that are needed. Please be clear and concise with any setup instructions.

An SWF video of your application using Jing would also be helpful but not required.

Your submission will be judged by a panel of peers based upon this criteria. If you would like to request an extension, please read Can challenge deadlines be extended for more time? on our FAQ. We do not want to grant extensions unless there is a valid reason.

ADDITIONAL INFO

Salesforce Web Services API Developer's Guide
Salesforce Bulk API Developer's Guide
Article: Loading Large Data Sets with the Force.com Bulk API
Sample code for a Java application using the Bulk API
Force.com Web Service Connector (WSC)
Java Command Line App Using the Salesforce WSC

PARTICIPATION


DISCUSSION

chiragmehta84
First thing - Really great challenge! Can I assume that each object will have one external id field viz., ExternalId__c - Text(18). This field will be helpful when doing upsert and also when loading parent child data (based on parent externalid field). Ok?
chiragmehta84, Oct 30, 2011 at 06:02 AM reply
jeffdonthemic
Yes! Assume there is some field that will be a unique ID for upserting.
jeffdonthemic (admin), Oct 31, 2011 at 06:01 AM
chiragmehta84
ok. thanks!
chiragmehta84, Dec 18, 2011 at 06:01 AM
elukaweski
will the source and target objects all be custom objects or does this program also need to process standard objects?
elukaweski, Nov 01, 2011 at 06:02 AM reply
jeffdonthemic
It needs to be able to custom object and most standard objects where possible. HTH
jeffdonthemic (admin), Nov 02, 2011 at 06:02 AM
kaushikganguly1986
Success and Error Files of the Upsert Operation are also required to be stored somewhere or they are not required?
kaushikganguly1986, Nov 01, 2011 at 06:02 AM reply
kaushikganguly1986
Success and Error Files of the Upsert Operation are also required to be stored somewhere or they are not required?
kaushikganguly1986, Nov 01, 2011 at 06:02 AM reply
jeffdonthemic
That would be a great feature!!!
jeffdonthemic (admin), Nov 02, 2011 at 06:02 AM
kaushikganguly1986
Thanks....
kaushikganguly1986, Nov 01, 2011 at 06:02 AM reply
forcebewithyou
Hey What is the size of data here? Are you looking at a Bulk API or Web Service API solution?
forcebewithyou, Nov 03, 2011 at 06:01 AM reply
jeffdonthemic
I would assume fairly large maybe100K records or more? My guess is that this might be a better with the Bulk API.
jeffdonthemic (admin), Nov 04, 2011 at 06:01 AM
kaushikganguly1986
Will the encrypted passwords which will be required for the conf file provided by the end user Or should it be taken care internally within the code?
kaushikganguly1986, Nov 04, 2011 at 06:01 AM reply
jeffdonthemic
Great question! I would say as of right now not to worry about encryption unless you really want to implement it. You should be able to store the plain text password in the config file. We may implement a solution to encrypt them later.
jeffdonthemic (admin), Nov 05, 2011 at 06:01 AM
kaushikganguly1986
Thanks!!!
kaushikganguly1986, Nov 05, 2011 at 06:01 AM
kaushikganguly1986
Well, while running the Data Loader in batch mode from the command line, we must encrypt the password. So are we looking at CLI? If yes, i think password encryption will be necessary
kaushikganguly1986, Nov 05, 2011 at 06:01 AM
sgurumurthy
Doing some basic groundwork on the challenge. Turns out while there is lot of documentation on insert, update & upsert, relatively little documentation on the query. Even the Java WSC project does not have all the required methods to query objects. So if we want to use Java we will have to rely on the native URL connection. Also wanted to point out an errata on page 73 of the bulk API developers guide. The URL to get query results is given as https://instance.salesforce.com/services/async/23.0/job/jobId/batchId/result/resultId. It is missing "batch" in between jobId and batchID. So the correct URL would be https://instance.salesforce.com/services/async/23.0/job/jobId/batch/batchId/result/resultId
sgurumurthy, Nov 04, 2011 at 06:01 AM reply
jeffdonthemic
Thanks for the info and helping out!
jeffdonthemic (admin), Nov 05, 2011 at 06:01 AM
avidev9
"n number of fields to be obfuscated " does that mean i can set first n records to that value or there is some condition?
avidev9, Nov 05, 2011 at 06:01 AM reply
jeffdonthemic
Thanks for the question. It means that fields to be obfuscated should not be hard coded in the app but configurable.
jeffdonthemic (admin), Nov 06, 2011 at 06:01 AM
kaushikganguly1986
Are we looking at CLI here? If yes, i think password encryption will be necessary. Any Suggestions please.
kaushikganguly1986, Nov 07, 2011 at 06:01 AM reply
sgurumurthy
@Kaushik, I think the focus is on the unload and load capabilities. Since it is a CLI interface it is plausible that in the interim a config file without password/token encryption would not be an issue.
sgurumurthy, Nov 07, 2011 at 06:01 AM reply
elukaweski
The one thing that is missing in the Bulk API is a call to retrieve all the outstanding jobs...would be nice to have for clean up after a test.
elukaweski, Nov 09, 2011 at 06:01 AM reply
elukaweski
This is my first submission....simple question. how and/or where do I upload my submission?
elukaweski, Nov 11, 2011 at 06:01 AM reply
sgurumurthy
@elukaweski, I see you have registered for the challenge as you are in the registered members list. You should see a button "Submit Your Entry" right above all these comments. Click this button, provide some comments, select your submission zip and hit the Submit button. Good Luck on your first submission.
sgurumurthy, Nov 11, 2011 at 06:01 AM reply
elukaweski
This is my first submission....simple question. how and/or where do I upload my submission?
elukaweski, Nov 11, 2011 at 06:01 AM reply
elukaweski
ok...got it...thanks
elukaweski, Nov 11, 2011 at 06:01 AM reply
elukaweski
uh...one more question....when I hit the submit button, will it allow me to upload more than one file?
elukaweski, Nov 11, 2011 at 06:01 AM reply
hwhitacre
Combine multiple files into a single zip, and upload the Zip file. This makes it easier for the person(s) reviewing your submission, and ensures that all of your pieces will stay together.
hwhitacre (admin), Nov 12, 2011 at 06:01 AM
forcebewithyou
Did the submission date just got postponed?
forcebewithyou, Nov 12, 2011 at 06:01 AM reply
jeffdonthemic
Yes! We had some technical issues with the site and extended the deadline. More info is on the blog: http://blog.cloudspokes.com/2011/11/extended-challenge-deadlines.html
jeffdonthemic (admin), Nov 13, 2011 at 06:01 AM
forcebewithyou
Oh! Thats great. Thank you.
forcebewithyou, Nov 13, 2011 at 06:01 AM
elukaweski
I have the same question about the deadline
elukaweski, Nov 12, 2011 at 06:01 AM reply
elukaweski
deadline seems to be extended 36 hours. see http://blog.cloudspokes.com/2011/11/extended-challenge-deadlines.html
elukaweski, Nov 12, 2011 at 06:01 AM reply
elukaweski
deadline seems to be extended 36 hours. see http://blog.cloudspokes.com/2011/11/extended-challenge-deadlines.html
elukaweski, Nov 12, 2011 at 06:01 AM reply
xxff
The characteristics of $10 gift, quickly laid hands on it. http://www.aliexpress.com/fm-store/708019/211271279-503424165/Wholesale-Freeshipping-HOT-WJ-020-Blue-and-white- Confucius-Bookmark-send-the-teacher-overseas.html http://www.aliexpress.com/fm-store/408333/211303435-506054643/Free-shipping-SB-003-2011-latest-fashion-cool-M-table- Watches-neutral.html
xxff, Dec 18, 2011 at 06:01 AM reply

The discussion board is closed for this challenge.



PARTICIPATION

16 members participating with 16 registered

TECHNOLOGIES

Salesforce.com

WINNER SELECTED

Ended Nov 13, 2011 at 4:00 AM
Winner Announced on November 18, 2011
View the challenge results
Give us your feedback about this challenge.

PRIZES

Total: $3,500
1st $2500
2nd $1000

SHARE

Chrome Extension

Install the CloudSpokes Chrome Extension for quick and easy access to the latest CloudSpokes challenges, your payments, leaderboards and more!