Issue with batch operation on room loading process
Created by: windmemory
Recently we've encountered problems related to https://github.com/wechaty/wechaty-puppet-hostie/issues/31
From the tests we've done on the issue above, we know that large batch operations in a short period of time will trigger it, which will cause wechaty-puppet-hostie
to restart.
Based on our test, comparing with Room.findAll()
, Contact.findAll()
seldom triggers the problem. Then I dig into the code and found that:
// src/user/contact.ts L224-L245
const BATCH_SIZE = 16
let batchIndex = 0
const invalidDict: { [id: string]: true } = {}
while (batchIndex * BATCH_SIZE < contactList.length) {
const batchContactList = contactList.slice(
BATCH_SIZE * batchIndex,
BATCH_SIZE * (batchIndex + 1),
)
await Promise.all(
batchContactList.map(
c => c.ready()
.catch(e => {
log.error('Contact', 'findAll() contact.ready() exception: %s', e.message)
invalidDict[c.id] = true
}),
),
)
batchIndex++
}
The code in the method Contact.findAll()
is doing a pagination call, which will reduce the count of requests made simultaneously, thus avoid the wechaty-puppet-hostie
issue.
But the code in Room.findAll()
does not have the pagination mechanism. So I think we should add pagination in the method.
We've tested the pagination in our scoped npm package @windmemory/wechaty
, and seems all working as expected.