{"id":18,"date":"2016-05-28T11:36:38","date_gmt":"2016-05-28T11:36:38","guid":{"rendered":"https:\/\/blogs.scummvm.org\/tkachov\/?p=18"},"modified":"2022-05-24T11:38:40","modified_gmt":"2022-05-24T11:38:40","slug":"gsoc-file-downloading-week","status":"publish","type":"post","link":"https:\/\/blogs.scummvm.org\/tkachov\/2016\/05\/28\/gsoc-file-downloading-week\/","title":{"rendered":"GSoC: File downloading week"},"content":{"rendered":"
\n

<\/p>\n

According to\u00a0my\u00a0proposal\u2019s schedule, I\u00a0should\u2019ve been working on\u00a0Storage interface, token saving and Dropbox and OneDrive stubs this week. I\u2019ve tried to\u00a0schedule less work before the midterm because I\u2019ve got some exams to\u00a0pass. Still, I\u00a0managed to\u00a0do\u00a0most of\u00a0that first week plan with my\u00a0preparation work. So, I\u2019ve finished that by\u00a0Tuesday and started working on\u00a0the things planned for the second week: files downloading.<\/p>\n

I\u2019ve also implemented DownloadRequest class that Tuesday. It\u00a0just reads bytes from NetworkReadStream and writes them into DumpFile.<\/p>\n

There was some small problem with DumpFile: it\u00a0fopen\u2019s a\u00a0file (creating\u00a0it, if\u00a0it\u00a0doesn\u2019t exist), but it\u00a0assumes that the file is\u00a0located in\u00a0some existing directory. When you\u2019re downloading a\u00a0file from the cloud into your local folder, you might not have the same directory hierarchy there. Thus, fopen fails and no\u00a0download occurs. As\u00a0there was no\u00a0createDirectory() method in\u00a0ScummVM, I\u00a0had to\u00a0implement one. So, now DumpFile accepts a\u00a0bool parameter, which indicates whether all directories should be\u00a0created before fopen\u2019ing the file.<\/p>\n

Then we\u2019ve decided\u00a0I should \u00abupgrade\u00bb our Requests\/ConnectionManager system by\u00a0adding RequestInfo struct and giving each Request an\u00a0id. User could use an\u00a0id\u00a0to\u00a0locate the Request through ConnMan and ask it\u00a0to\u00a0change its state. That\u2019s mostly needed when we\u00a0want to\u00a0make the same request again: for example, if\u00a0some error occurred, we\u00a0probably would like to\u00a0try again in\u00a0a\u00a0few seconds. This system added RETRY state for Requests, so\u00a0user\u2019s code can easily ask to\u00a0retry Request. The original idea with RequestInfo and ids was rethought, so\u00a0now we\u2019re actually using pointers to\u00a0Request instances. That way one can easily affect Request: cancel, pause or restart it with its methods.<\/p>\n

With that system\u00a0I could easily implement the OneDriveTokenRefresher. That might be\u00a0not the best class name, but it\u00a0describes it\u00a0quite well =) OneDrive access tokens expire after an\u00a0hour, so\u00a0that\u2019s very sad to\u00a0get an\u00a0error because you\u2019re working with the app more than an\u00a0hour. This class wraps the usual CurlJsonRequest (which is\u00a0used to\u00a0receive JSON representation of\u00a0server\u2019s response) to\u00a0hide errors regarding expired token. Basically, it\u00a0just peeks into received JSON and, if\u00a0there is\u00a0an\u00a0error message, refreshes the token and then tries to\u00a0do\u00a0the original request again, using the new token. With that new system, it\u00a0just pauses the original request and then retries it\u00a0\u2014 no\u00a0need to\u00a0save the original request\u2019s parameters somewhere because they are just stored along with the paused request.<\/p>\n

After I\u00a0finished that useful token refreshing class, I\u2019ve implemented OneDriveStorage::download(), thus completing the second week plan too. The plan also has \u00abauto detection\u00bb feature, which is\u00a0being delayed a\u00a0little because it\u00a0would be\u00a0called from the GUI and I\u2019m not working on\u00a0the GUI yet. Yet, there was no\u00a0folder downloading in\u00a0my\u00a0plan, and it\u2019s obviously a\u00a0must have feature, so\u00a0that\u2019s what\u00a0I did next. The FolderDownloadRequest uses Storage\u2019s listDirectory() and download() methods, so\u00a0it\u2019s easy to\u00a0implement and it\u00a0works with all Storage implementations which have these methods working.<\/p>\n

I\u00a0actually implemented OneDriveStorage\u2019s listDirectory() a\u00a0little bit later because there is\u00a0no\u00a0way to\u00a0list directories recursively in\u00a0OneDrive\u2019s API. There is\u00a0such opportunity in\u00a0Dropbox, so\u00a0it\u00a0takes me\u00a0one API call to\u00a0list whole directory and all its subdirectories contents. To\u00a0list OneDrive directories recursively, I\u00a0had to\u00a0write a\u00a0special OneDriveListDirectoryRequest class, which lists the directory with one API call, then lists each of\u00a0its subdirectories with other calls, then their subdirectories, and so\u00a0on\u00a0until it\u00a0lists it\u00a0all.<\/p>\n

I\u2019ve been asked to\u00a0draw some sequence diagrams explaining all these Requests\/ConnectionManager systems, so\u00a0that\u2019s what I\u2019m going to\u00a0do\u00a0now. I\u2019m not a\u00a0fan of\u00a0sequence diagrams and I\u2019m going to\u00a0draw them in\u00a0my\u00a0own style, which, I\u00a0believe, makes them simpler.<\/p>\n

UPD:<\/b>\u00a0here they come, two small diagrams (available on\u00a0ScummVM wiki<\/a>\u00a0too):<\/p>\n

<\/div>\n

File sync comes next week and\u00a0I guess I\u2019m going to\u00a0discuss that feature a\u00a0lot before\u00a0I actually start implementing\u00a0it.<\/p>\n<\/div>\n","protected":false},"excerpt":{"rendered":"

According to\u00a0my\u00a0proposal\u2019s schedule, I\u00a0should\u2019ve been working on\u00a0Storage interface, token saving and Dropbox and OneDrive stubs this week. I\u2019ve tried to\u00a0schedule less work before the midterm because I\u2019ve got some exams to\u00a0pass. Still, I\u00a0managed to\u00a0do\u00a0most of\u00a0that first week plan with my\u00a0preparation work. So, I\u2019ve finished that by\u00a0Tuesday and started working on\u00a0the things planned for the second […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-18","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/blogs.scummvm.org\/tkachov\/wp-json\/wp\/v2\/posts\/18","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blogs.scummvm.org\/tkachov\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blogs.scummvm.org\/tkachov\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blogs.scummvm.org\/tkachov\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blogs.scummvm.org\/tkachov\/wp-json\/wp\/v2\/comments?post=18"}],"version-history":[{"count":1,"href":"https:\/\/blogs.scummvm.org\/tkachov\/wp-json\/wp\/v2\/posts\/18\/revisions"}],"predecessor-version":[{"id":22,"href":"https:\/\/blogs.scummvm.org\/tkachov\/wp-json\/wp\/v2\/posts\/18\/revisions\/22"}],"wp:attachment":[{"href":"https:\/\/blogs.scummvm.org\/tkachov\/wp-json\/wp\/v2\/media?parent=18"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blogs.scummvm.org\/tkachov\/wp-json\/wp\/v2\/categories?post=18"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blogs.scummvm.org\/tkachov\/wp-json\/wp\/v2\/tags?post=18"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}