I have a question about our very old, very tall River Birch.
If the root of your River Birch was cut right next to the trunk and it was a major root and was cut six years ago when the dropping leaves problem first occurred, then cutting the root may have something to do with the problem. If the cutting for your fence occurred recently, then it is not the source of the problem. River Birches tolerate a lot of rain water, so it is not the source of the problem.
If the "flakes" are pieces of bark, it is normal for birches to drop pieces of bark.
If the top of the tree has lost most of its leaves and side growth is excessive, the problem may be the Bronze Birch Borer. D shaped holes and ridges along the trunk are signs of the borer. Cut into one of the ridges to see if it is packed with sawdust. Pruning and spraying by an arborist can save the tree.
Other birch problems include scale, fungus and insects, but are not as frequently found as the Birch Borer.