Salesforce 2 Salesforce "Data Dumper"

OVERVIEW

CODE

One of our frequent tasks at CloudSpokes is to spin up Force.com Developer orgs for challenge participants to use during a specific challenge. We typically use ANT to install code and configurations but importing data is always a chore. We want to build a tool that runs on a local machine or in the cloud that takes data from a source salesforce.com org and dumps it into target salesforce org.

The tool will allow you to specify multiple objects, the fields to query for each object and the credentials for both the source and target orgs. The records would always be upserted in the target org. We also want to have the ability to obfuscate specific field data. For instance, when dumping data from production to a developer org, we typically change email addresses to something like “noemail@cloudspokes.com”.

We currently use roughly the same process to move data from SQL Server to our database.com org on a nightly basis. The code is in Java and you can find it on this GitHub repo. It should make a great starter app.

REQUIREMENTS

Use the Force.com Web Services API or Bulk API to transfer data (see Additional Info section for relevant docs). We this this is a great job for the Bulk API.

Use any language that you’d like to build this tool however there is a lot of sample code in Java. The tool does not need to have any sort of UI; we’d prefer to run it from the command line. The tool will read a config/xml file which contains all data for a particular job including but not limited to:

- source connection info
- target connection info
- object/table with data -- must support multiple objects
- fields (or query)
- where clause
- external id
- n number of fields to be obfuscated and the value (email__c => ‘noemail@cloudspokes.com’, phone => ‘111-222-3333’)

When the process runs it will read the config/xml file, query the source org for data and push the data into the target org. You will need to allow data to be upserted using externalIds. Assume that both the source and target orgs have the same data structure for the objects. All records will always be upserted into the target org.

Submission
Upload all your source code as a zip and provide any documentation and/or instructions that are needed. Please be clear and concise with any setup instructions.

An SWF video of your application using Jing would also be helpful but not required.

SUBMISSION DETAILS

Upload all your source code as a zip and provide any documentation and/or instructions that are needed. Please be clear and concise with any setup instructions.

An SWF video of your application using Jing would also be helpful but not required.

Your submission will be judged by a panel of peers based upon this criteria. If you would like to request an extension, please read Can challenge deadlines be extended for more time? on our FAQ. We do not want to grant extensions unless there is a valid reason.

ADDITIONAL INFO

Salesforce Web Services API Developer's Guide
Salesforce Bulk API Developer's Guide
Article: Loading Large Data Sets with the Force.com Bulk API
Sample code for a Java application using the Bulk API
Force.com Web Service Connector (WSC)
Java Command Line App Using the Salesforce WSC

PARTICIPATION

DISCUSSION

First thing - Really great challenge! Can I assume that each object will have one external id field viz., ExternalId__c - Text(18). This field will be helpful when doing upsert and also when loading parent child data (based on parent externalid field). Ok?

chiragmehta84 , Oct 30, 2011 at 06:02 AM reply

Yes! Assume there is some field that will be a unique ID for upserting.

jeffdonthemic (admin), Oct 31, 2011 at 06:01 AM

ok. thanks!

chiragmehta84 , Dec 18, 2011 at 06:01 AM

will the source and target objects all be custom objects or does this program also need to process standard objects?

elukaweski , Nov 01, 2011 at 06:02 AM reply

It needs to be able to custom object and most standard objects where possible. HTH

jeffdonthemic (admin), Nov 02, 2011 at 06:02 AM

Success and Error Files of the Upsert Operation are also required to be stored somewhere or they are not required?

kaushikganguly1986 , Nov 01, 2011 at 06:02 AM reply

Success and Error Files of the Upsert Operation are also required to be stored somewhere or they are not required?

kaushikganguly1986 , Nov 01, 2011 at 06:02 AM reply

That would be a great feature!!!

jeffdonthemic (admin), Nov 02, 2011 at 06:02 AM

Thanks....

kaushikganguly1986 , Nov 01, 2011 at 06:02 AM reply

Hey What is the size of data here? Are you looking at a Bulk API or Web Service API solution?

forcebewithyou , Nov 03, 2011 at 06:01 AM reply

I would assume fairly large maybe100K records or more? My guess is that this might be a better with the Bulk API.

jeffdonthemic (admin), Nov 04, 2011 at 06:01 AM

Will the encrypted passwords which will be required for the conf file provided by the end user Or should it be taken care internally within the code?

kaushikganguly1986 , Nov 04, 2011 at 06:01 AM reply

Well, while running the Data Loader in batch mode from the command line, we must encrypt the password. So are we looking at CLI? If yes, i think password encryption will be necessary

kaushikganguly1986 , Nov 05, 2011 at 06:01 AM

Thanks!!!

kaushikganguly1986 , Nov 05, 2011 at 06:01 AM

Great question! I would say as of right now not to worry about encryption unless you really want to implement it. You should be able to store the plain text password in the config file. We may implement a solution to encrypt them later.

jeffdonthemic (admin), Nov 05, 2011 at 06:01 AM

Doing some basic groundwork on the challenge. Turns out while there is lot of documentation on insert, update & upsert, relatively little documentation on the query. Even the Java WSC project does not have all the required methods to query objects. So if we want to use Java we will have to rely on the native URL connection. Also wanted to point out an errata on page 73 of the bulk API developers guide. The URL to get query results is given as https://instance.salesforce.com/services/async/23.0/job/jobId/batchId/result/resultId. It is missing "batch" in between jobId and batchID. So the correct URL would be https://instance.salesforce.com/services/async/23.0/job/jobId/batch/batchId/result/resultId

sgurumurthy , Nov 04, 2011 at 06:01 AM reply

Thanks for the info and helping out!

jeffdonthemic (admin), Nov 05, 2011 at 06:01 AM

"n number of fields to be obfuscated " does that mean i can set first n records to that value or there is some condition?

avidev9 , Nov 05, 2011 at 06:01 AM reply

Thanks for the question. It means that fields to be obfuscated should not be hard coded in the app but configurable.

jeffdonthemic (admin), Nov 06, 2011 at 06:01 AM

Are we looking at CLI here? If yes, i think password encryption will be necessary. Any Suggestions please.

kaushikganguly1986 , Nov 07, 2011 at 06:01 AM reply

@Kaushik, I think the focus is on the unload and load capabilities. Since it is a CLI interface it is plausible that in the interim a config file without password/token encryption would not be an issue.

sgurumurthy , Nov 07, 2011 at 06:01 AM reply

The one thing that is missing in the Bulk API is a call to retrieve all the outstanding jobs...would be nice to have for clean up after a test.

elukaweski , Nov 09, 2011 at 06:01 AM reply

This is my first submission....simple question. how and/or where do I upload my submission?

elukaweski , Nov 11, 2011 at 06:01 AM reply

@elukaweski, I see you have registered for the challenge as you are in the registered members list. You should see a button "Submit Your Entry" right above all these comments. Click this button, provide some comments, select your submission zip and hit the Submit button. Good Luck on your first submission.

sgurumurthy , Nov 11, 2011 at 06:01 AM reply

This is my first submission....simple question. how and/or where do I upload my submission?

elukaweski , Nov 11, 2011 at 06:01 AM reply

ok...got it...thanks

elukaweski , Nov 11, 2011 at 06:01 AM reply

uh...one more question....when I hit the submit button, will it allow me to upload more than one file?

elukaweski , Nov 11, 2011 at 06:01 AM reply

Combine multiple files into a single zip, and upload the Zip file. This makes it easier for the person(s) reviewing your submission, and ensures that all of your pieces will stay together.

hwhitacre (admin), Nov 12, 2011 at 06:01 AM

Did the submission date just got postponed?

forcebewithyou , Nov 12, 2011 at 06:01 AM reply

Yes! We had some technical issues with the site and extended the deadline. More info is on the blog: http://blog.cloudspokes.com/2011/11/extended-challenge-deadlines.html

jeffdonthemic (admin), Nov 13, 2011 at 06:01 AM

Oh! Thats great. Thank you.

forcebewithyou , Nov 13, 2011 at 06:01 AM

I have the same question about the deadline

elukaweski , Nov 12, 2011 at 06:01 AM reply

deadline seems to be extended 36 hours. see http://blog.cloudspokes.com/2011/11/extended-challenge-deadlines.html

elukaweski , Nov 12, 2011 at 06:01 AM reply

deadline seems to be extended 36 hours. see http://blog.cloudspokes.com/2011/11/extended-challenge-deadlines.html

elukaweski , Nov 12, 2011 at 06:01 AM reply

The characteristics of $10 gift, quickly laid hands on it. http://www.aliexpress.com/fm-store/708019/211271279-503424165/Wholesale-Freeshipping-HOT-WJ-020-Blue-and-white- Confucius-Bookmark-send-the-teacher-overseas.html http://www.aliexpress.com/fm-store/408333/211303435-506054643/Free-shipping-SB-003-2011-latest-fashion-cool-M-table- Watches-neutral.html

xxff , Dec 18, 2011 at 06:01 AM reply

The discussion board is closed for this challenge.

CHALLENGE DETAIL

Salesforce 2 Salesforce "Data Dumper"

OVERVIEW

REQUIREMENTS

SUBMISSION DETAILS

ADDITIONAL INFO

PARTICIPATION

DISCUSSION

PARTICIPATION

TECHNOLOGIES

WINNER SELECTED

PRIZES

SHARE

Chrome Extension

MAIN LINKS

MEMBER LINKS

SPOKES

PLATFORMS