{"id":14,"date":"2016-05-12T11:35:04","date_gmt":"2016-05-12T11:35:04","guid":{"rendered":"https:\/\/blogs.scummvm.org\/tkachov\/?p=14"},"modified":"2022-05-24T11:35:40","modified_gmt":"2022-05-24T11:35:40","slug":"gsoc-cloud-integration-preparation-work","status":"publish","type":"post","link":"https:\/\/blogs.scummvm.org\/tkachov\/2016\/05\/12\/gsoc-cloud-integration-preparation-work\/","title":{"rendered":"GSoC: Cloud Integration Preparation Work"},"content":{"rendered":"<p>After discussing API design, I\u2019ve done some research on\u00a0Dropbox, Google Drive and OneDrive API and discussed further steps with Peter and Eugene. I\u2019ve identified some preparation steps I\u00a0should\u00a0do\u00a0before GSoC\u00a0starts. Doing this preparation work would also get me\u00a0familiar with ScummVM\u00a0code, coding &amp;\u00a0formatting conventions and commit guidelines.<\/p>\n<p>I\u2019m working on\u00a0it\u00a0in\u00a0my\u00a0public\u00a0<a href=\"https:\/\/github.com\/Tkachov\/scummvm\">fork<\/a>\u00a0of\u00a0scummvm repo on\u00a0Github right in\u00a0the master branch. Eugene helped me\u00a0with configure (which I\u2019m not using yet, as\u00a0I\u2019ve started with MSVC\u00a0here) and watches how I\u2019m doing.<\/p>\n<p>My preparation work plan includes the following tasks:<\/p>\n<ul>\n<li>integrating JSON parser;<\/li>\n<li>writing API skeleton;<\/li>\n<li>adding libcurl and writing some simple wrapper for\u00a0it;<\/li>\n<li>adding some cloud-related configs keys, so\u00a0ScummVM would remember my\u00a0access token and\u00a0I won\u2019t have to\u00a0authenticate every time\u00a0I launch\u00a0it.<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<p>We\u2019ve decided that\u00a0<a href=\"https:\/\/github.com\/MJPA\/SimpleJSON\">SimpleJSON<\/a>\u00a0is\u00a0a\u00a0good library to\u00a0use as\u00a0JSON parser in\u00a0ScummVM. It\u2019s really quite simple library, which consists of\u00a0two classes and uses\u00a0C++ standard library. It is now available as\u00a0Common::JSON in my fork and uses ScummVM\u00a0classes.<\/p>\n<p>A\u00a0few weeks back, I\u00a0still was thinking that all cloud-related work would be\u00a0done within separate thread spawned in\u00a0main(). But when I\u2019ve decided to\u00a0actually add such thread (so\u00a0my\u00a0JSON examples could work and not delay ScummVM\u2019s launcher), I\u00a0understood that the original idea to\u00a0use libcurl\u2019s blocking functions is\u00a0a\u00a0bad idea. libcurl\u2019s functions block thread execution until the request is\u00a0complete. Because of\u00a0that, our cloud thread won\u2019t react to\u00a0the user\u2019s commands when it\u2019s \u00abbusy\u00bb. And that means if\u00a0you started downloading 2\u00a0GB on\u00a056\u00a0kbps, you can\u2019t cancel it\u00a0without killing ScummVM\u00a0process!<\/p>\n<p>That\u2019s totally not what we\u00a0want. There is\u00a0no\u00a0direct thread spawning in\u00a0ScummVM and even though TimerManager actually runs callbacks in\u00a0separate threads, such threads might work even when ScummVM main thread is\u00a0finished (and TimerManager could be\u00a0implemented so\u00a0it\u00a0won\u2019t be\u00a0running callbacks in\u00a0separate threads anymore). So, I\u00a0had to\u00a0think of\u00a0something. At\u00a0first my\u00a0idea was to\u00a0add a\u00a0callback through TimerManager, which would be\u00a0executed every second. It\u00a0would\u2019ve checked whether the user wanted ScummVM to\u00a0make some API requests and whether these requests were complete. But most of\u00a0the time we\u00a0don\u2019t work with cloud, and thus such callback would be\u00a0called even though it\u00a0was unnecessary.<\/p>\n<p>Now the idea is\u00a0a\u00a0bit more complex. We\u00a0would have to\u00a0use async libcurl functions, because we\u00a0don\u2019t want our callback to\u00a0block. So, we\u00a0can\u2019t make our API methods to\u00a0return the requested information, but instead those should receive some function pointer and call the passed function when this information is\u00a0ready. I\u00a0also believe that there could be\u00a0more than one request pending, and some methods would require more than one REST API request.<\/p>\n<p>Thus, I\u00a0think of\u00a0making special Request class, which would act as\u00a0a\u00a0base class for all methods implementations. For example, ListDirectoryRequest would be\u00a0not only requesting the first \u00abpage\u00bb of\u00a0directory contents, but make such requests until whole directory is\u00a0listed. When all the information is\u00a0gathered, it\u00a0would call the callback.<\/p>\n<p>Storage class, which would actually represent cloud storage API in\u00a0ScummVM, has all the API methods we\u00a0need. But these methods would be\u00a0creating new Request objects only. The implementation would be\u00a0hidden within these Request classes. So, Storage class would also contain some Request list. When it\u2019s empty, there is\u00a0nothing to\u00a0do\u00a0for Storage, and it\u00a0does nothing. But when it\u00a0is\u00a0not, Storage would start a\u00a0timer and poll these requests, whether they have complete what they were doing. When the request is\u00a0complete, it\u00a0is\u00a0removed from this list. When list gets empty, Storage stops the timer.<\/p>\n<p>Finally, there is\u00a0also a\u00a0CloudManager. This is\u00a0a\u00a0class, which would be\u00a0loading all connected storages from ScummVM configs and providing fast access to\u00a0the currently active Storage. So, the idea is\u00a0to\u00a0have all the storages loaded and ready, so\u00a0user can easily switch between them in\u00a0the Options menu. But only one Storage would be\u00a0\u00abactive\u00bb. This storage would be\u00a0used to\u00a0sync saves, upload and download files until user changes that. CloudManager would not only have a\u00a0getter for\u00a0it, but also a\u00a0few \u00abshortcut\u00bb methods, which would be\u00a0using current storage within. For example, you won\u2019t have to\u00a0do\u00a0<i>cloudManager-&gt;getCurrentStorage()-&gt;syncSaves()<\/i>. Instead, you can just do\u00a0<i>cloudManager-&gt;syncSaves()<\/i>.<\/p>\n<p>I\u00a0have not implemented Requests yet, but there are already some simple CloudManager and Storage stubs.<\/p>\n<p><b>UPD:<\/b>\u00a0I\u2019ve implemented\u00a0<a href=\"https:\/\/github.com\/Tkachov\/scummvm\/commit\/81b8930af7aa5eeb5c3768620595fe28cbd89adb\">first simple Request<\/a>\u00a0today. It\u2019s not doing anything useful, but it\u00a0shows that requests can \u00abwork\u00bb for a\u00a0few\u00a0<i>handler()<\/i>\u00a0calls and then stop. When no\u00a0requests are working, timer is\u00a0automatically stopped.<\/p>\n<p>Why do\u00a0we\u00a0need timer in\u00a0the first place? Well, that\u2019s because we\u00a0want to\u00a0know whether request is\u00a0complete and we\u00a0have to\u00a0poll\u00a0<a class=\"emphasis\" href=\"https:\/\/curl.haxx.se\/libcurl\/c\/curl_multi_info_read.html\">curl_multi_info_read<\/a>\u00a0to\u00a0do\u00a0that. Plus, this allows\u00a0us to\u00a0react to\u00a0user\u2019s commands. For example, user might want to\u00a0cancel the download, and then Request would be\u00a0stopped on\u00a0the next timer tick.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>After discussing API design, I\u2019ve done some research on\u00a0Dropbox, Google Drive and OneDrive API and discussed further steps with Peter and Eugene. I\u2019ve identified some preparation steps I\u00a0should\u00a0do\u00a0before GSoC\u00a0starts. Doing this preparation work would also get me\u00a0familiar with ScummVM\u00a0code, coding &amp;\u00a0formatting conventions and commit guidelines. I\u2019m working on\u00a0it\u00a0in\u00a0my\u00a0public\u00a0fork\u00a0of\u00a0scummvm repo on\u00a0Github right in\u00a0the master branch. Eugene [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-14","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/blogs.scummvm.org\/tkachov\/wp-json\/wp\/v2\/posts\/14","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blogs.scummvm.org\/tkachov\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blogs.scummvm.org\/tkachov\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blogs.scummvm.org\/tkachov\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blogs.scummvm.org\/tkachov\/wp-json\/wp\/v2\/comments?post=14"}],"version-history":[{"count":1,"href":"https:\/\/blogs.scummvm.org\/tkachov\/wp-json\/wp\/v2\/posts\/14\/revisions"}],"predecessor-version":[{"id":15,"href":"https:\/\/blogs.scummvm.org\/tkachov\/wp-json\/wp\/v2\/posts\/14\/revisions\/15"}],"wp:attachment":[{"href":"https:\/\/blogs.scummvm.org\/tkachov\/wp-json\/wp\/v2\/media?parent=14"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blogs.scummvm.org\/tkachov\/wp-json\/wp\/v2\/categories?post=14"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blogs.scummvm.org\/tkachov\/wp-json\/wp\/v2\/tags?post=14"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}