Machine Learning in C++

[spandan11106]

🏷️ Discussion Type

Question

Body

Hey everyone,

I’m currently building a C++ machine learning framework — nothing too serious, just a fun project to understand things like tensors, memory handling, and autograd at a deeper level. I’d really appreciate any suggestions on what features, ideas, or improvements I should explore next. Whether it’s design decisions, performance tricks, or even things to avoid, I’m open to all of it.

If anyone’s interested in contributing, discussing ideas, or just experimenting together, feel free to jump in. Always happy to learn from people who’ve worked on similar systems.

🔗 Repo: https://github.com/spandan11106/GradCore-Tensor.git

[amine-za]

Hi, this is a really interesting project, building tensors and autograd from scratch in C++ is a great way to understand ML internals.

One idea you could explore is adding a small benchmarking suite to compare performance of different tensor operations, especially memory allocation patterns. It could also be interesting to experiment with parallelization (like OpenMP) for basic ops.

From an infrastructure perspective, setting up CI (GitHub Actions) with tests and benchmarks could make it easier to track improvements over time.

Happy to follow the progress, looks like a great learning project.

[Vansh150705]

This is a really nice project, building tensors and autograd in C++ is a great way to understand how ML works internally.

Some simple ideas you can try next:

• Add basic operations like matrix multiplication and more element-wise functions
• Implement a few activation functions like ReLU and Sigmoid
• Try making a small neural network (like a simple linear model)

For performance:

• You can try using OpenMP or threads to speed up operations
• Also focus on reducing unnecessary memory copies

For improvements:

• Add some test cases to check if your tensor operations are correct
• Create small examples (like training on dummy data)
• Add a README section explaining how your system works

Overall, this is a great learning project and will help you understand ML at a deeper level. Keep going

[Gecko51]

Building autograd from scratch in C++ is a solid exercise. A few specific things worth exploring next:

Gradient checking — before adding more features, implement numerical gradient verification. Compare your autodiff result against (f(x + eps) - f(x - eps)) / (2 * eps) for a handful of test cases. It'll catch subtle bugs in backward passes that are otherwise almost impossible to find. This should be your first addition.

Topological sort for the backward pass — if your graph traversal isn't already sorted, you'll get wrong gradients on non-trivial computation graphs. Most autograd bugs trace back to this.

Expression templates (CRTP) — once you're happy with correctness, look into lazy evaluation. Keeping tensor ops lazy lets you fuse them before allocating, which removes a lot of temporary allocations in the hot path. Eigen uses this heavily and is worth reading for the pattern.

Arena allocator for temporaries — instead of new/delete for every intermediate tensor during a forward pass, a bump allocator you reset between iterations is much faster and simpler than a general allocator.

One thing to avoid: don't try to optimize before you have a working end-to-end training loop. Get XOR working with a 2-layer MLP first. If your gradients are right there, they're probably right everywhere. Then profile before guessing where the bottlenecks are.