Best practices - synchronize database with external source?

Pigbenis

Molten Core Raider
679
279
I have a bit of a technical issue that I'm working through that I'd like some advice on. I have an external source for data, it's a JSON string and I need to store the info in my database. The problem is this data can change - new elements can be added, modified, or removed. Whenever these changes happen I need to update the data in my database. This is Java, btw. Code has been simplified for easy reading.

This seems to handle creating and updating. What I haven't figured out is what if an element is deleted from the JSON. How do I tell if it no longer exists and then remove it from my database?

At the start I could get a list of all the elements currently in the database. As I'm looping through the JSON nodes add them to another list. Once i'm done adding new ones loop through both lists and remove elements not found in the first list. I'm just really trying to avoid this second loop. Anyone have any pointers?
 

Izo

Tranny Chaser
18,530
21,403
23272659.jpg

I donno, what database is it you're importing too? Maybe there is a tool for it you can use as inspiration?
Random google video:
 

Desidero

N00b
163
2
Something like this should work well enough if you aren't dealing with a lot of data:

HashMap<key,name> current - store all keys currently in the database
HashMap<key,name> updates - track updates
HashMap<key,name> newEntries - track new entries

 

Desidero

N00b
163
2
Ah. I had it set up that way so you could do more efficient batch inserts/updates/deletes, but that works too.
 

Pigbenis

Molten Core Raider
679
279
Ah. I had it set up that way so you could do more efficient batch inserts/updates/deletes, but that works too.
Humm interesting. My ORM package has a batch function, but I'm dealing with a small number of transactions (~10) at a time. It is on Android, though. I may look into it.