javascript - PouchDB about 3x slower inserting 4800 documents over websql -

been exploring use of pouch db potentially leverage offline sync instead of doing our own. have cordova app pulls data rest api , saves web sql store. part of around 5k physical locations various stores. timed on cordova app, , download stores & save them, along request project information in 11 seconds. saving of these 5k records documents in pouch db takes 30 seconds. not counting request time.

here's do:

let db = pouchproxy.getdb(); this.render("loading", { message: "saving locations" }); // stores api let stores = res.stores; let docs = []; // check saved stores, checking ids start store_ db.alldocs({ include_docs: true, startkey: 'store_', endkey: 'store_\uffff' }).then((results) => {   // go through saved stores   (let row of results.rows) {     let doc = row.doc;     // number id. _id 'store_idnumfromapi'     let id = parsefloat(doc._id.split('store_')[1]);     if (isnan(id)) {       throw "cannot nan";     }     // iterate through stores api     (var = 0; < stores.length; i++) {       let store = stores[i];       // find store , local record       if (store.id === id) {         // sets _id , other properties store api object new object         let map = pouchproxy.storetodocmap(store);         // set revision         map._rev = doc._rev;         docs.push(map);         stores.splice(i, 1);         break;       }     }   }    // go through remaining stores , push docs set   (let store of stores) {     docs.push(pouchproxy.storetodocmap(store));   }    // save things   console.log(date.now());   return db.bulkdocs(docs); }).then(() => {   // second time stamp use 30 seconds   console.log(date.now());   // calculate geolocation stuff   (let store of docs) {     store.distance = this.distancebetweenpoints(store, { lat: position.coords.latitude, long: position.coords.longitude });   }   docs.sort((a, b) => {     return b.distance - a.distance;   });   this.render("store-list", { stores: docs.slice(0, 19) }); }).catch(function (err) {   console.log(err); }); 

hope code clear enough. thought switching 1 document stores, feel individual lookups more difficult , expensive.

edit, revised code. performance worse :(

as per suggestion, broke down list of 5000 chunks. played couple different sizes. 300 & 500, both had similar times saving data. here's how looks now:

savelocations(db, stores) { // position   var storeslices = [];   stores.sort(function (a, b) {     return a.id - b.id;   });   stores.eachslice(300, (slice) => {     storeslices.push(slice);   });    console.log(date.now());   this.commitbatch(db, storeslices, 0, () => {     console.log(date.now());     // this.sortandrenderstores(docs, position);     this.render("store-list", { stores: [] });   }); }  commitbatch(db, slices, index, callback) {   let stores = slices[index];   db.alldocs({ include_docs: true, startkey: 'store_' + stores[0].id, endkey: 'store_' + stores[stores.length - 1].id }).then((results) => {     let docs = [];     (let row of results.rows) {       let doc = row.doc;       let id = parsefloat(doc._id.split('store_')[1]);       if (isnan(id)) {         throw "cannot nan";       }       // iterate through stores api       (var = 0; < stores.length; i++) {         let store = stores[i];         // find store , local record         if (store.id === id) {           let map = pouchproxy.storetodocmap(store);           // set revision           map._rev = doc._rev;           docs.push(map);           stores.splice(i, 1);           break;         }       }     }      // go through remaining stores , push docs set     (let store of stores) {       docs.push(pouchproxy.storetodocmap(store));     }      db.bulkdocs(docs).then(() => {       index++;       if (index < slices.length) {         this.commitbatch(db, slices, index, callback);       }       else {         callback();       }     });   }).catch(function (err) {     console.log(err);   }); } 

my guess issue here reading documents memory in 1 go (or @ least, every document prefixed store_, alldocs() doing), , writing of documents in 1 go well.

5000 documents lot. , experience pouchdb there sweet spot bulkdocs() (depending on document size), , way less 5000. applies both indexeddb , websql.

most speed code batching in chunks of 100-500, while paginating in alldocs() using limit , startkey (i.e. using strategies described in this blog post), , inserting 100-500 @ time using bulkdocs().

as proof i'm not fibbing, can check out http://npm-browser.com/, inserts 500 metadata documents npm per batch in single bulkdocs(), , doesn't take 7 seconds per batch. (most of time spent waiting on network download documents.)