Interesting thing happened during the weekend.
Dr Han started a project to get everybody interested in putting up a site that tackle 2019 nCov.
See https://www.facebook.com/drhanlau/posts/2775625775829452 for more information.
I took up scraping stuff from the web and one thing that helped was the code below:
from pandas.io.json import json_normalize
import json, requests
import dateparser
from datetime import datetime
import sqlalchemy as sa
e=sa.create_engine('mysql://the dsn here')
r=requests.get('some_url_that+returns_json')
d=json.loads(r.content)
df=json_normalize(d) #specify the right keys
df.to_sql('temp_table', e, if_exists="append", index=False)
with e.begin() as cnx:
insert_sql = 'INSERT IGNORE INTO the_table (SELECT * FROM temp_table)'
cnx.execute(insert_sql)
#the_table should have a field set as unique to avoid duplicates
If the source site returns xml, get xmltodict, convert the xml to json and do the above.
It was an exhilarating experience, face paced, and getting to work with many Heroes in their own fields.
Thumbs up!
Comments !