Personal computing discussed

Moderators: renee, SecretSquirrel, just brew it!

 
Crayon Shin Chan
Minister of Gerbil Affairs
Topic Author
Posts: 2313
Joined: Fri Sep 06, 2002 11:14 am
Location: Malaysia
Contact:

Need advice on good coding conventions

Thu Jul 26, 2018 7:42 pm

Recently I've been working on a rather complex project, the Quantum Resistant Ledger, and I find myself asking questions like these really often:

1. Should I serialize the data used in a class before passing it down to the database to save, or should I let the database serialize the object, so that I have to pass less information about the object to the database?
2. They say you should write unit tests first. But, that assumes you know that your design/implementation is the best one for the problem already. What if you don't know what's the best way and you need to try out several ways of coding the solution? Thus, writing unit tests first is just being idealistic, isn't it?
3. Where and to what extent the abstraction of dirty details should start. For example, we use LevelDB, which has this 'batch' method of coalescing operations. How can I know, without wasting a lot of time coding every approach out, in which layer I can start abstracting that away without making it harder for classes above to do things?
4. They say your unit tests should mock out everything else. But when I do that, two things happen: the test gets monstrously complex because of all the setup and mocking that needs to happen, and when I'm done writing it, I still can't be sure it actually works when you connect it with real classes - and when the other classes change, somebody has to remember that this unittest, which mocks the other classes' interfaces, needs to be changed too. Which might not be obvious, and in all likelihood, might not happen.

Classes that wrap around data like Protobuf are the ones hardest to mock away, because the class which I'm supposed to test relies a lot on its methods, and I have to mock them all out and provide reasonable mock values for them. And what if they change in the future, and somebody forgot that this class uses them? I'd be testing against a totally useless, mocked up interface instead of the real thing.


I have no idea who to talk to about problems like these. Surely others must have figured out best practices for situations like these already. What should I even put into Google? The closest I've found are "Coding conventions".

Are there any good books that deliver timeless advice on these topics?
Mothership: FX-8350, 12GB DDR3, M5A99X EVO, MSI GTX 1070 Sea Hawk, Crucial MX500 500GB
Supply ship: [email protected], 12GB DDR3, M4A88TD-V EVO/USB3
Corsair: Thinkpad X230
 
Pancake
Gerbil First Class
Posts: 161
Joined: Mon Sep 19, 2011 2:04 am

Re: Need advice on good coding conventions

Thu Jul 26, 2018 9:46 pm

Crayon Shin Chan wrote:
Recently I've been working on a rather complex project, the Quantum Resistant Ledger, and I find myself asking questions like these really often:

1. Should I serialize the data used in a class before passing it down to the database to save, or should I let the database serialize the object, so that I have to pass less information about the object to the database?
2. They say you should write unit tests first. But, that assumes you know that your design/implementation is the best one for the problem already. What if you don't know what's the best way and you need to try out several ways of coding the solution? Thus, writing unit tests first is just being idealistic, isn't it?
3. Where and to what extent the abstraction of dirty details should start. For example, we use LevelDB, which has this 'batch' method of coalescing operations. How can I know, without wasting a lot of time coding every approach out, in which layer I can start abstracting that away without making it harder for classes above to do things?
4. They say your unit tests should mock out everything else. But when I do that, two things happen: the test gets monstrously complex because of all the setup and mocking that needs to happen, and when I'm done writing it, I still can't be sure it actually works when you connect it with real classes - and when the other classes change, somebody has to remember that this unittest, which mocks the other classes' interfaces, needs to be changed too. Which might not be obvious, and in all likelihood, might not happen.

Classes that wrap around data like Protobuf are the ones hardest to mock away, because the class which I'm supposed to test relies a lot on its methods, and I have to mock them all out and provide reasonable mock values for them. And what if they change in the future, and somebody forgot that this class uses them? I'd be testing against a totally useless, mocked up interface instead of the real thing.


I have no idea who to talk to about problems like these. Surely others must have figured out best practices for situations like these already. What should I even put into Google? The closest I've found are "Coding conventions".

Are there any good books that deliver timeless advice on these topics?


Object-oriented programming is a methodology I've used for the last 20 years or so. It's a good way to break down complexity into manageable chunks - or objects with well-specified behaviours. Then you need only worry about the interactions between different objects. Event propagation and handling is another important thing to decouple complex interacting objects and simplify design.

But none of what I know and use is "cool" or "hip"...
 
Ryhadar
Gerbil XP
Posts: 463
Joined: Tue Oct 21, 2008 9:51 pm

Re: Need advice on good coding conventions

Thu Jul 26, 2018 9:56 pm

I'm a bit sick at the moment, so apologies if this is too long winded/doesn't make sense. Hopefully this helps. I come from the C#.NET/MS SQL world so I might use terminology that doesn't jive with your lexicon. Here it goes...

1. Should I serialize the data used in a class before passing it down to the database to save, or should I let the database serialize the object, so that I have to pass less information about the object to the database?

Best answer I have to this one is that it depends. Is it trivial to implement a custom object in your database or does it take a bit of work? Is this work extensible or would you have to make something custom for every call to the database? For example, in MS SQL Server adding support for custom objects that you specify in .NET is a bit tricky when you're working with an data abstraction framework like Entity Framework. In that case, it was just easier and it was understood by the other developers on the team to serialize the data or just represent the data as indepedent parameters to a stored procedure call. All things being equal I would say use an object if you can as it's easier to read. But databases aren't exactly object oriented anyway so honestly I wouldn't lose much sleep over it. When in doubt, follow the standards set by the team.

2. They say you should write unit tests first. But, that assumes you know that your design/implementation is the best one for the problem already. What if you don't know what's the best way and you need to try out several ways of coding the solution? Thus, writing unit tests first is just being idealistic, isn't it?


Test driven development says that you should start out your coding writing a test that will fail, then re-work your program so you get a passing result. When you're just starting your work, usually this is just a call to an object that doesn't exist. Then you write code to make sure your unit tests pass. Rather. Rinse. Repeat.

Maybe that is idealistic, but they also say that you should have how you want to write your program out already in the form of pseudo-code to prevent such U-turns -- and that might just be too idealistic too! From my personal experience it's just easier to write your tests when you're coding. I find it more difficult to come back to hundreds of lines of code then write unit tests from scratch off of my finished work. A lot of times too you're testing some pretty common things in your program like null object checks, divide by zero checks, etc. Stuff that even if you take a hard about face in your program you're probably going to reuse.

If when you're writing your code you change directions enough that you break compatibility with what you've already written just delete the unit test. I think some developer shops frown on this so you can also just decorate your unit test as "deprecated" if your test framework allows for that. Just make sure you commit often to your source control repository branch so you can always go back when you decide your first attempt was better. :)

3. Where and to what extent the abstraction of dirty details should start. For example, we use LevelDB, which has this 'batch' method of coalescing operations. How can I know, without wasting a lot of time coding every approach out, in which layer I can start abstracting that away without making it harder for classes above to do things?

Also kind of depends on the development house. Some developers say to wedge this in your DB model (your table schema) layer. I think that's kind gross since I like my model layer to just be plain old class objects so I like having a data service layer in between your model and application layers. However, I've definitely put such calls in model layers before for applications where it didn't really make much sense to tack on a whole extra layer (small applications that do straightforward CRUD operations and not much else) or when there wasn't enough time to retrofit a service layer.

As an example, if I had a table called "Stuff" with Id, Name, and Date fields I'd have a DB model object that matched that schema. From there I would have a base service layer class that handled CRUD operations or other batch operations that needed to happen in addition. All data service layers would inherit from this class. Most classes would probably just inherit the base class and not do much else. If special consideration needed to be done to certain entities I'd inherit the base class and override the behavior of certain methods. For example, if when adding to the table "Stuff" I also needed to make sure I add a history record to a table called "StuffHistory" my add method would probably look like:
1) Begin transaction
2) INSERT INTO Stuff
3) Use Id created from Stuff, INSERT INTO StuffHistory
4) Commit transaction

For other tables, like "Users" maybe just inserting a record into the table is sufficient. So it would follow the default behavior specified by the base class.

You can also do some cool things with your model layer like implementing from a common interface. For example, I used to work on a project where every entity in the database needed the same audit fields (last update date, create date, creating user, and updating user). Every entity in our model layer needed to implement this interface to be used in our service layer. So, for example, since every entity had a create user field everytime we inserted a record we used the application's user context to fill out this field in our service layer base class. No need to write it out every time. Similarly, on UPDATE calls we made sure NOT to update the create user field.

4. They say your unit tests should mock out everything else. But when I do that, two things happen: the test gets monstrously complex because of all the setup and mocking that needs to happen, and when I'm done writing it, I still can't be sure it actually works when you connect it with real classes - and when the other classes change, somebody has to remember that this unittest, which mocks the other classes' interfaces, needs to be changed too. Which might not be obvious, and in all likelihood, might not happen.

Correct. You should be reducing your dependencies in your code so that when you unit test you can shove in pretty much any scenario you like. And indeed, this can get very complex but if you allow for dependency injection it can go a lot smoother when you mock. It's definitely annoying though. Almost to the point where you're supporting two code bases: the application and your unit testing helper code. :roll:

If you can, try and spend some time mocking your database table classes with some generic data. You can and probably will be doing this by hand, but there are some mocking frameworks that help with this (if you work with C#, moq is really nice). You'll essentially be making your own, tiny in memory database right into your application. What I like to do, is have a core set of data of common scenarios as a private member of my unit test class and refresh this data back to the default before every test method run (your testing framework should have some means to do this). Then, if I want to test out a certain condition, such as maybe making sure my code doesn't add a "Name" string longer than 50 characters to a table, I'll modify the value in my unit test and make sure it's caught in my code as expected.

As for not being sure if it works when connecting with real classes... well, yeah. I mean, your job as a developer is to come up with the scenarios that will lead to undesirable behavior. That means that your application should handle issues with the database isn't available, or when the you've been denied permissions to a resource. If you have QA team members they're great resources to poke for ideas since their job is to break your code. :D If you're worried about missing scenarios then I'm sorry to say but you're going to have misses. It's a part of life unfortunately. But if you're working on a good team your colleagues will point them out to you in code review time. If you wanted to go the extra mile you could also setup some integration tests as part of your test suite as well. Integration tests actually use real resources so usually you need dedicated hardware for that.

To answer you question about making sure that the next person to come along needs to know about updating the unit test you're absolutely right. But that's their job. They should be adding unit tests to account for their code updates and if you're reviewing someone else's code it's your job to call them out if they don't. In addition, since developers hate writing unit tests they're going to go looking for existing ones. I promise you. If you're worried about false positives unfortunately they can happen. Unit testing isn't perfect, especially since they're only as good as the tester that implements them. Best advice I can give here is to give the next developer that comes along a hand: document your code (even if it's just in the comments) and make sure your unit tests are as atomic as possible. Follow the AAA pattern: arrange, act, assert: https://msdn.microsoft.com/en-us/librar ... x#Anchor_3

And also make sure to check for null objects a lot.:D
 
Crayon Shin Chan
Minister of Gerbil Affairs
Topic Author
Posts: 2313
Joined: Fri Sep 06, 2002 11:14 am
Location: Malaysia
Contact:

Re: Need advice on good coding conventions

Fri Jul 27, 2018 1:56 pm

Man, thanks a lot! You helped me realize that a lot of the rules out there are 'you don't have to follow them all the time" type of rules.
Mothership: FX-8350, 12GB DDR3, M5A99X EVO, MSI GTX 1070 Sea Hawk, Crucial MX500 500GB
Supply ship: [email protected], 12GB DDR3, M4A88TD-V EVO/USB3
Corsair: Thinkpad X230

Who is online

Users browsing this forum: No registered users and 1 guest
GZIP: On