Transpiling private member fields with Babel

Posted on May 22, 2021 by Siva

The current project I am working on deals with a lot of JavaScript. It is actually 100% TypeScript. JavaScript execution is made very fast in modern browsers, especially the V8 engine, thanks to technologies like hidden classes. JavaScript is a bolt-on language with humble roots to a modern scripting language thanks to all the ES proposals that eventually became standards. One such proposal is private fields. The product I am working on requires heavy computation on the front-end and every bit of optimization counts. The good thing about private fields is that they allow browsers to push the optimization boundaries as unlike the regular fields, private fields have to be declared before they can be used. This essentially means the optimizer can make decisions based on static code rather than deferring it to runtime and adjusting it on the fly as needed like how it’s done with the hidden classes.

While the private fields are very promising, as of this writing only Chrome has production support, Firefox support is almost there with the functionality currently hidden behind configuration (otherwise, it throws the error “Uncaught SyntaxError: private fields are not currently supported”). Safari has just started private fields support from 14.1. That means, for the time being the code needs to be transpiled to support older browsers or Firefox. The transpilation is possible with tools such as Babel.

I recently found esbuild that can transpile and bundle code and is much faster. However, I needed to integrate the code as a component into Docusaurus documentation. Docusaurus comes with its own build process and internally uses WebPack and Babel. So, for this specific use case I am stuck with Babel. However, the actual setup was TypeScript code compiled by tsc, transpiled and bundled by esbuild and finally the resulting bundle is further transpiled and bundled through Babel and WebPack.

Docusaurus has two modes of building, one for development (or should we say, documentation) while the other is for production. The development mode worked just fine but the production mode was resulting in runtime errors related to private field not being present and that’s where the problems stared. On careful observation of the stack trace, I identified the problem that is a combination of several things.

  1. The way esbuild (and Babel) transpile private fields to older targets is by using WeakMaps. Then at runtime, read/writes are checked against the WeakMaps with the object as the key and each private field has its own WeakMap.
  2. In order to provide access to the private variables, corresponding property getter/setters are defined.
  3. Occasionally, a Child class has to override these setter/getters and use the super keyword to delegate to the Parent class.
  4. Babel has an option called loose and if it’s specified as an option to the preset, then any plugin which supports that option is also provided the same value.
  5. Docusaurus uses loose=true and as a result all the class transformations are being done loosely, so to speak.
  6. One such transformation is the super which became a.prototype. Hence, super.xyz becomes t.prototype.xyz where t is the variable name given to this in that context.
  7. This is where the problem is. The esbuild code has logic while accessing the private field to check for presence in the WeakMap and ideally the look up should be this but in this case, it is t.prototype and hence it doesn’t find the field and thinks that the code is trying to access a private field that is not owned by the object.

After a bit of trail and error, I managed to solve this by specifying loose as false for the plugin “@babel/plugin-transform-classes” and everything started working as expected. Of course, by being strict with the transformations there is a lot of performance penalty on top of already losing performance in not using the private fields itself.

Obviously, correctness has higher priority to performance. So, for the time being, the documentation will support all the browsers with slightly lower performance and this situation will be reevaluated in the future when private fields become widespread.

Posted in JavaScript | Tagged , , | Leave a comment

Stripe Integration on Google App Engine with Golang

Posted on September 13, 2019 by Siva

Google App Engine went GA for Go 1.12 runtimes and there are some key changes with it. The old appengine packages are not expected to work and instead everyone is expected to migrate to the Goolge Cloud client library. For long time existing apps this may be a pain but it’s in the right direction to make the code as standard as possible for better portability both within and outside GCP. Below is an example of getting rid of a custom way of integrating as a result of moving to a standard way of developing your app.

The Stripe Go library on their GitHub documentation explicitly have a section on using it within Google App Engine. It went along the lines of “If you’re running the client in a Google AppEngine environment, you’ll need to create a per-request Stripe client since the http.DefaultClient is not available. Here’s a sample handler:”

This is because, up until Go 1.9, the only way to do external http calls from the server were using the urlfetch package. This required passing a context to create the http client. And the context had to be an app engine context which was only available as part of a http request. But with Go 1.12, there is no app engine context, there is no urlfetch (Go 1.11 allowed both ways). Instead, you can use the standard context package and create a Background context and use http.Client for making http requests. As a result of all of this, it is now possible to create the Stripe Client API object only once and use it across many requests. Given the Client.API object is very heavy to initialize (it creates lots and lots of objects one for each type of API even though you only use a handful), creating it once and reusing it with each request is much better. Given http.Client is inherently thread-safe, everything should work just fine just as expected if the app was being developed in a standard way.

So while moving from legacy code is a pain, I am all for the changes that make the app be a standard go app with built-in web server.

Lastly, I almost decided to create the client API object at the time of initializing the instance but decided against it as the best practice is to do lazy loading of less frequently used stuff. So, instead I resorted to the sync.Once pattern that would initialize the API object on the first usage of Stripe call from the app for the running instance.

Posted in Google App Engine, Stripe | Leave a comment

Selenium Test Automation Using A Free Monad

Posted on August 17, 2019 by Siva

I have been working on a project that ended up with a lot of UI and as time passed I realized there is no way I can ensure the quality of the code without test automation. One may say, no one should be writing code for serious production use without having test automation. I agree. It’s just that there is one person doing all the things, there is only so much that can be accomplished. Eventually, the code base stabilized enough that I was confident to go down the automation route. Selenium IDE turned out to be the simplest way of doing UI automation. It was so easy I ended up automating a lot of the flows. All it required was just executing the flow and recording it and practically no code involved.

Except for a few issues, everything was smooth with Selenium IDE. The main issues were a) reusing of the test fragments as if they were function calls so the same test fragment could be executed with different parameters (as far as I can tell there is no way to do this), b) any sort of mass editing is not possible unless it’s manually done directly in the underlying JSON. These are acceptable limitations but I felt that I could have done so much more automation easily if I had these options at my disposal.

After a few weeks, I had to do some major change to the UI in certain flows. That’s when the pain of modifying the tests became evident. With Selenium IDE, on one hand you can easily open up a “.side” file and pick a specific test suite and execute all the tests within it and watch the UI to ensure nothing is broken after a major set of code changes. On the other, if something broke, doing surgery to the already recorded test script felt a bit painful. Again, nothing that a normal user can’t tolerate or anything like that. But for me it all came to speed of changing tests, increasing test coverage and ensuring the stability of tests. This last part of ensuring stability in fact requires post processing a recorded test by carefully sprinkling wait commands.

So while I enjoyed the Selenium IDE and the script files authored with it for the simplicity of easily being able to replay and watch for any bugs, I wanted more control on the authoring process. All searches pointed to writing direct WebDriver code using the underlying API in Java. I didn’t want to pursue that route.

In this backdrop, I was also trying to enhance my understanding of functional programming and one of the things I was learning about was Free Monads. I found the article “What does free buy us?” very helpful in understanding the Free Monad concept. I realized that may be I could solve my Selenium IDE script authoring issue using the Free Monad. How nice it would be to take all the Selenium IDE commands and represent them as a Functor of sum of all the commands and lift them into Free Monad, write tests using the handy “do” notation of Haskell, compose them, reuse with different parameters and at the end, have an interpreter that simply spits out the “side” format which is a simple json format?

This idea of using Free Monad to author tests that can be simply built by composition in Haskell was too tempting that I practically set aside my main project for a few days and worked on this. The end result, I have a nice library and all my tests moved from “side” format to “hs” code (Haskell). Test authoring is a breeze, test coverage increased significantly as I could author tests as monadic functions and pass different parameters and then just “map” over a list of parameters to do all kinds of test coverage.

Below is a small snippet of test code in Haskell using the ESDL I created.

testEmailFbrFree = test "fbr-email-create-free" "Create Free Feedback Request - Email Test" $
   do
     runTest "Login"
     click $ routeHas "Request Feedback"
     click $ routeHas "Create"
     forM_ fbps $ uncurry enterFeedbackProvider
     submitFbr
   where fbps = map (id &&& (++"@example.com")) ["A","B","C","D","E"]

As can be seen, I could use monad constructs like forM_, arrow operators like &&& and all the fun of composing code in Haskell and the amazing fact that all of this eventually as a Free Monad results in a “container” that is peeled one piece at a time and interpreted to create the final JSON Selenium IDE file is a wonderful learning exercise for me.

Posted in Haskell | Leave a comment

Twisted type inference

Posted on July 27, 2019 by Siva

Recently I was reading an article about finally tagless interpreters and came across a piece of code that at first glance was totally confusing. Those who are familiar with Haskell types and classes should be able to follow the code below even if they don’t know anything about interpreters.

  1. Let’s define a class

class A a where f :: Int -> a

2. Now let’s define a type

data A1 = A1 Int

It’s going to “implement” the class, so

instance A A1 where f x = A1 x

3. Now let’s define a second type

newtype A2 = A2 Int

It’s going to also “implement” the instance, so

instance A A2 where f x = A2 x.

Yes, A1 and A2 are implementing the same but what matters is that they are different types. I also purposely made one data and the other newtype but that doesn’t really matter.

4. Now, let’s define functions that take A1 and A2 as arguments.

showA1 (A1 x) = “showing A1 “++show x

showA2 (A2 x) = “showing A2 “++show x

So far nothing strange. Following both statements will perfectly work

showA1 (f 5)

showA2 (f 5)

Because of return type polymorphism, the expression (f 5) will correctly evaluate to f belonging to A1 or A2 appropriately based on the type inference from the show functions.

The following code,

let e = (f 5) in (showA1 e, showA2 e)

will, however, not work. This is because, the variable “e” needs to resolve to either A1 or A2 types and can’t be both at the same time.

How can the above problem be solved so that we don’t have to write the (f 5) twice? This is important because, (f 5) could in theory be a large program all composed of the various functions from the class A and we don’t want to write duplicate code as not only is it laborious but also can go out of sync.

Return polymorphism again to rescue by using the following clever technique. Let’s define another instance of class A as follows

instance (A r1, A r2) => A (r1, r2) where f x = (f x, fx)

What is happening above is, the (f x) on the left hand side is providing a return type polymorphism where the return type of A (r1, r2) where r1 and r2 are themselves instances of A. This is essentially letting a single term to be “cloned” into two terms but with different types. Now, the above code to use both showA1 and showA2 together can be written as

let (e1,e2) = (f 5) in (showA1 e1, showA2 e2)

and the (f 5) will clone and bind the same term to e1 and e2 respectively as two different typed expressions.

If you use Arrows, the above can be simple and sweet as (showA1 *** showA2) (f 5) and (f 5) automatically gets inferred to be operating in the context of a pair which further using return type polymorphism invokes the A (r1,r2) instance’s f code.

In case you are wondering why all this fuss, think of showA1 and showA2 as two separate interpreters for the same expression of your DSL implemented via the class A. The above technique allows having more than one interpreter operate on the expression written once (but represented multiple times at runtime each as the appropriate type).

Posted in Uncategorized | Tagged , | Leave a comment

Concurrency, reflection and marshalling

Posted on January 5, 2019 by Siva

Last few months I have been working on a highly concurrent blockchain system. We choose Go language as it has some nice language level constructs to deal with concurrency. The overall experience of using Go for the project had been very pleasant because of the fast compilation time and server startup time. These two along make it very productive for the development team.

There was one nagging issue that took a while to figure out. The protocol requires collecting a set of unique tickets. These tickets were stored as an array within a struct representing the block (of the block chain). Occasionally, we see the block being received to contain duplicate tickets. This should never happen. We took care that this never happens with a mutex as tickets can be added concurrently (very rarely).

It turns out the reason for the duplicates is a combination of concurrency, reflection and marshalling. That is, while the struct is getting marshalled, the slice is getting simultaneously updated resulting in this problem. The reason it took a while to figure this out is, the modification is done in a such a way that it doesn’t preserve the order of the original tickets when merging another set of tickets.

If you look at the code in Msgpack, at

https://github.com/vmihailenco/msgpack/blob/master/encode_slice.go

it first gets the length of the array and iterates over that. So, while the iteration is happening, even if the array grows, it will only marshall up to the initial length. That’s not the problem. The real problem is, the reflection on a struct field always returns the latest value in the field. So, even if the array field has been changed, the reflected value will always gives the array field of the struct and the Index function will return the latest value in the index. Obviously this is the right behavior as reflection shouldn’t be working off of copies of data as of a call to Value or Field.

So, what was the fix? Ensuring that the merging happens by only appending new tickets and not disturbing the order of existing tickets. It was OK to marshall only a subset of tickets based on the length at the time of calling the encoding function.

One interesting aspect of this is that when dealing with marshalling, it’s practically not possible to do fine grained locking to the field level since the marshalling is for the enclosed object (immediate or multiple levels above).

Posted in Uncategorized | Leave a comment

What is safe for url could be unsafe from security standpoint

Posted on April 27, 2018 by Siva

Google Datastore comes with a way to encode the key of the entity that is URL safe. This may sound strange at first that a database key api should care about URLs. Since the datastore supports nested data model where an entity can only exist in the context of another, the key of an entity is it’s own key part and the key of it’s parent (which itself is key part + it’s own parent’s key …). It’s like a folder structure /a/b/c. The actual format of the key is the entity name and the int/string id. So, a key can look like this

/User,1234/Mortgage,3394/Payment,7839

This means, there is a payment entity 7839 that’s child of Mortgage entity 3394 that is child of the User 1234. Since this format uses the forward slash, ‘/’, which is used extensively in URLs, using the key as is will have problems. That’s why there are Key.Encode and datastore.DecodeKey(string) methods.

In an app I am working on, the 3rd level entity (user being the first level) can be seen by anyone but not the second level entity. Only the user who is parent of the 2nd level entity can see it. Initially I almost didn’t bother about security just making assumption that the keys are quite random and hard to guess. Then it hit me that when a user has access to the 3rd level key, it’s possible to decode it and construct the 2nd level key and then use the URL that fetches 2nd level entity data (as it would for that person’s own data) and pass the 2nd level encoded key constructed by decoding the 3rd level key. Luckily, I already have the API execute in the context of a valid session and so all I had to do is something like (in golang)

if *entity.GetKey().Parent() != *user.GetKey() {

return error.Error(“You are not authorized to view this data”)

}

With that, the app is secure!

 

Posted in Uncategorized | Leave a comment

Supporting 3D Secure Card Payments With Stripe Checkout

Posted on April 22, 2018 by Siva

I have been experimenting on building a self-service app that accepts transaction based payments. I was looking into Paypal and Stripe and at the moment started off with Stripe. Stripe has a simple checkout that launches their own popup, accepts the payment data and submits it to their servers and returns a token (all happening in the browser) and then this token can be submitted from the browser to your server code. The server code would take the token and does the actual charge. Apart from I had to do the integration into a VueJS app and load the dependent js files dynamically only when a user wants to pay, the integration was pretty smooth. Best thing is that Stripe website has some cards for testing the integration which comes very handy.

As I was testing the various cards listed, I saw sample 3D Secure cards and they haven’t worked with my integration. I got an error that the card got declined. So, I read further documentation on 3d Secure cards and how the integration works. Stripe has something called Elements which allows you to embed the payment UI right into your website (but still secure and you won’t know your customer data) and they have an API for 3d secure cards based on the elements. However, given I already did the token based checkout integration (which is actually very simple and intuitive for end users), I didn’t wanted to start over. I managed to do the integration and the rest of the post is about how I did it.

Thing is, not all cards are 3d secure cards and even those that are, not all require going through the 3d secure payment process. So, it’s important to be able to detect when that flow is needed and when it’s not. Unfortunately, the card object given in the token payload from the regular checkout process doesn’t have this information on whether it needs the 3d secure process or not. To over come this, rather than directly submitting the token payload to the sever in the token call back, we can first create a source using the token.id. This returns a source which contains a card with more details and one of them is whether the three_d_secure is required or not. If it’s not required, just submitting the source payload to the server (instead of the token payload) should be sufficient given that the server side charge seamlessly works with token id or source id (of course, you need to make sure the server side code covers all these cases in a generic manner).

Assuming the three_d_secure is required, then we need to create another secure 3d source from the first source. This will be the 3rd call from browser to stripe (token -> source -> secure source). As part of this 3rd call, we also need to specify a return url. Unlike the previous flows that rely on a callback being made from the current browser page to the server, the 3d secure payment process requires first redirecting to a 3rd party where the payment details can be confirmed and after that a redirect is done to the browser. The redirect url can contain any context information that is needed to tie back to the product/service being purchased.

The return URL provides the 3d secure source id, client secret and livemode parameters. This 3d secure source id can be passed to the charge API like the token id or the first source id obtained from the token. Assuming the user went through the 3d secure payment flow and confirmed the payment, the server side API will succeed. Otherwise the same card declined error will happen.

Stripe even has a 3d secure card payment test page. Interestingly, the page has both to accept the payment and to cancel and in both cases the same callback URL with the same parameters seems to be passed. This makes it not know whether the user went through the confirmation process or not and have to issue the server side charge call to figure out what happened.

Note that, even when a card requires 3d secure process, the redirect object can indicate redirect.status as succeeded which means we don’t need to go through the redirect process. This is probably a decision either the card issuer or Stripe (with it’s ML) making that decision to reduce friction and enhance UX.

Overall, this exercise made me to learn a bit more about the online payment integration and the different types of payment flows. Hope this write up helps anything trying to do something similar.

Posted in Uncategorized | 1 Comment

Recursion, Monad and Laziness

Posted on September 1, 2017 by Siva

I am always excited about being able to write software that is easy to read and easy to maintain. Haskell is supposed to provide a lot of greatness and so I have been trying to enhance my knowledge of Haskell but more importantly all the concepts that comes with this beautiful language. I have a program that runs pretty well and I tried to do some refactoring (the more you learn the Haskell idioms, then more you hate your initial code :)). After refactoring, the code went into infinite loop and I was stumped. I was counting on the laziness and the functor features to keep the code terse.

The situation is that the original function wasn’t recursive (only 1 level) before refactoring. However, to reduce the complexity of if/else for the caller, I made it indirectly recursive (a -> b -> a) in one case. Further, I relied on the laziness so that it won’t be recursive forever without explicit base case handling. But when I had a pure function c that is lifted into the b which is an IO monad, I had the problem. By rewriting the c function itself work directly with monads solved the problem.

Posted in Haskell | Leave a comment

Mom, Dad or Alexa?

Posted on October 5, 2016 by Siva

How to engage with 6 to 10 month babies? One thing that usually works is to take them near a switch and keep turning on and off and showing them how the light turns on/off because of that. I can’t exactly tell whether they understand it, but it sure is a way to stimulate their little brains.

A few months back I had to use Amazon Alexa at work to do prototype a custom skill. Subsequently I ended up buying one for home. Then I learned that Alexa can be used to do home automation and so bought a few LIFX bulbs. With that, now I can turn on a light in my living room by saying “Alexa, turn on the living room light”. Of course, Alexa can be used for a few other things as well, such as “Alexa, play some music”, “Alexa, tell some kids jokes” and so on. We have a couple of Alarms in the morning to get my son ready to school and so when the alarm rings we say “Alexa, stop”.

So, the more we use Alexa, the more we keep saying out “Alexa”. Our 5 and half month daughter has been observing these and so she knows there is something about Alexa. She smiles (consistently so it’s not a random act) when we say “Alexa” and the blue ring of light glows.

With that context, today I got back to my turning the switch on/off experiment with a modern twist. I used Alexa instead of a regular wall switch to show my daughter how the light goes on and off. Not sure how much she got it yet, but will keep trying it.

I am wondering if the first word she is going to say will be “Alexa” instead of Mom or Dad.

Posted in Uncategorized | Leave a comment

Scratch that thought

Posted on July 14, 2016 by Siva

These days everyone talks about teaching programming to kids. In fact yesterday I came across an article by some guy mentioning about a kickstarter project for a robot to learn coding for kids and his thoughts on that.  That is prompting me to take a little time of my busy life to write some stuff (I think it’s long due).

My son who recently turned 7 wants to program. It’s not because I wanted him to learn. He got to see someone else at home using Scratch for a class when he was 4 yr old. So he would drag and drop a few commands and make the cat go round in circles and make it meow and is happy. I even took him to two Scratch coding events that are meant for 8 yr and above kids. The intention is not to get him to learn much sooner but just for him to have fun and a different experience than the regular routine.

Apart from the MIT Scratch, he also has a few programs on iPad. The one I like the best for kids is the Lego Mindstorms fix the factory app. Even though what one does with it is actually coding it doesn’t feel like that at all. It’s as if the kid is trying to figure out what he needs to make the robot do as a series of commands. Of course, deep down every program is ultimately a set of commands and so it is a program, but doesn’t feel like it.

Then there is Kodable which I actually like and we got the paid version. Another one he uses is Hopscotch. I actually saw how he did with Kodable but other than that I don’t really pay much attention to his “programming” explorations. In fact, the other day he was checking out scratch website for Minecraft games online :). Nothing wrong with that, that’s what he should be doing at that age. Not immerse in coding. I am going to write more about kids learning programming.

I want to actually get back to what I wanted to write in this post. It actually may serve to validate what I want to say later.

Professionally I am interested in “computation” and the best way to articulate a piece of computation. After seeing my son struggle with scratch for anything non-trivial, I actually spent time learning Scratch myself. I did it for two reasons. One to understand the implications of scratch like environment in enterprise applications and the other to understand the complexity of computations that a non-technical person can articulate using Scratch like programs.

So, I took some notes as I saw my son trying to do some stuff and also myself when trying to solve some puzzles using Scratch. Here are my thoughts on Scratch.

1) Event driven without “wait” is difficult to get it right. It’s easy to use “Broadcast and wait” but not “clone myself” because the “clone myself” doesn’t come with an equivalent “wait” construct. So, when trying to create lots of clones with state variables changing in the loop, the “wait” has to be simulated explicitly by setting a variable and waiting till the cloning is complete.
2) Inserting blocks is fine, but when trying to readjust the sequence of blocks for logic or move them around, it can easily cause confusion and leads to mistakes. And they will be hard to debug.
3) When the complexity reaches a certain level, say two level of loops or more than 12 or so blocks, it starts to feel like writing program like a developer
4) Scratch doesn’t have function return values. For developers who are familiar with a lot of advanced concepts, programming in scratch actually feels like a burden.
5) Sprite specific variables should have been automatically prefixed so it’s easy for novice developers to readily know the difference.
6) Seems like there is no way to loop through all the sprites that have been instantiated. This forces to sometime think of the sprite specific data as variables sometimes and lists other times.
7) The fact that there is no way to refer to specific sprite also make the broadcast receiving a bit more complicated. It requires setting up an identity to each sprite via a sprite specific variable, setting the identify value in a global variable in a context that is broadcasting and then comparing it in the receiving event.
8) The grouping of various constructs like Motion, Looks, etc is a bit confusing. This is not actually a bad thing to do but having an “advanced” mode where the developers can just search and find would be good. For example, I may know I want the “broadcast” but don’t have to remember what type of a block it is. This is especially important if there are lots of domain specific blocks.
9) Once the code gets to a state where some level of debugging is required, the debugging experience is more or less the same as directly writing the code. Of course, it’s possible to watch variables and things like that, but still complexity of doing additional code review and pin-pointing what went wrong where is similar to using a programming language.
10) It’s easy to share normal programming code on the forums and things like that. But sharing scratch would require taking screenshots and sharing. This also means, other people interested in helping out would need to construct the code and then provide the fix or improvement.

Posted in Uncategorized | Tagged , | Leave a comment