{"paper_id":"3065cd7f-2c3c-4544-8a4f-8c165defa0b2","body_text":"Efficient Federated Learning for distributed\nNeuroImaging Data\nBishal Thapaliya †,1,3,∗, Riyasat Ohib †,1,2, Eloy Geenjar 1,2, Jingyu Liu 1,3,\nVince Calhoun 1,2,3, Sergey Plis 1,3\n1TReNDS Center, Atlanta, Georgia, USA\n2 Georgia Institute of T echnology , Atlanta, Georgia, USA\n3 Georgia State University , Atlanta, Georgia, USA\nCorrespondence*:\nBishal Thapaliya\nbthapaliya16@gmail.com\nEqual Contribution:\n†These authors contributed equally to this work and share first authorship\nABSTRACT\nRecent advancements in neuroimaging have led to greater data sharing among the scientific\ncommunity. However, institutions frequently maintain control over their data, citing concerns\nrelated to research culture, privacy, and accountability. This creates a demand for innovative\ntools capable of analyzing amalgamated datasets without the need to transfer actual data\nbetween entities. To address this challenge, we propose a decentralized sparse federated\nlearning (FL) strategy. This approach emphasizes local training of sparse models to facilitate\nefficient communication within such frameworks. By capitalizing on model sparsity and selectively\nsharing parameters between client sites during the training phase, our method significantly lowers\ncommunication overheads. This advantage becomes increasingly pronounced when dealing\nwith larger models and accommodating the diverse resource capabilities of various sites. We\ndemonstrate the effectiveness of our approach through the application to the Adolescent Brain\nCognitive Development (ABCD) dataset.\nKeywords: Efficient Federated Learning, Neuro-imaging, Sparsity\n1 INTRODUCTION\nDeep learning has transformed fields like computer vision, natural language processing, and is also starting\nto transform the field of neuroimaging. As deep learning models grow, distributed and collaborative training\nbecomes essential, especially when sensitive data is spread across distant sites. Collaborative MRI data\nanalysis offers profound insights, allowing researchers to utilize data beyond a study’s original scope.\nAs MRI scans are often preserved, vast amounts of data accumulate across decentralized research sites.\nTechnological advancements have increased data complexity and reduced costs, prompting researchers to\nharness data from various groups for larger sample sizes and revealing significant features while preserving\nthe privacy of the data.\nTraining models on more data, while preserving data privacy is thus crucial. Aggregating data from\ndifferent sources to a central server for training can however expose this sensitive information, raising\nethical concerns. Federated learning (FL) addresses this by allowing devices or organizations to train\nmodels locally and share training details without sharing the actual data.\n1\n.CC-BY 4.0 International licensemade available under a \n(which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is \nThe copyright holder for this preprintthis version posted May 15, 2024. ; https://doi.org/10.1101/2024.05.14.594167doi: bioRxiv preprint \n\nBishal et al. Sparse FL for NeuroImaging\nIn federated learning (FL), a central server coordinates training, and client sites communicate only model\nparameters, keeping local data private. In the decentralized setting, the server usually doesn’t exist and\nclients train a model collaboratively among themselves. However, challenges arise due to data’s statistical\nheterogeneity, limited communication bandwidth, and computational costs. In this work, we focus on\naddressing the communication efficiency inherent in distributed federated neuro-imaging pipelines through\nthe training of sparse models at local sites.\n1.1 Federated Learning\nFederated Learning distinguishes itself from traditional distributed learning approaches, such as those\ndiscussed by McMahan et al. (2017), through its unique characteristics:\n1. Non-IID Data: The training data across different clients are not identically distributed, which means\nthat the data at each local site may not accurately represent the overall population distribution.\n2. Unbalanced Data: The amount of data varies significantly across clients, leading to imbalances in\ndata representation.\n3. Massive Distribution: Often, the number of clients exceeds the average number of samples per client,\nillustrating the scale of distribution.\n4. Limited Communication: Communication is infrequent, either among clients in a decentralized\nsetting or between clients and the server in a centralized setting, due to slow and expensive connections.\nOne of the main focuses of this work is to reduce the communication costs between the server and\nclients in a centralized setting or among clients in a decentralized setting when dealing with non-IID and\nunbalanced data. This is achieved by identifying a sub-network based on the data distributions at each\nlocal site and transmitting only the parameters of this sub-network in each communication round r. In\neach round, a fixed set of ˜K clients is sampled from all K clients, and federated training continues on the\nselected sub-network of those clients. The general federated optimization problem encountered is detailed\nnext.\n1.2 Federated Optimization Problem\nIn the general federated learning (FL) setting, a central server tries to find a global statistical model by\nperiodically communicating with a set of clients. The federated averaging algorithm proposed by McMahan\net al. (2017); Koneˇcn`y et al. (2016); Bonawitz et al. (2019) is applicable to any finite sum objective of the\nform\nmin\nw∈Rd\nf(w), where f(w) = 1\nn\nnX\ni=1\nfi(w). (1)\nIn a typical machine learning problem, the objective function fi(w) = ℓ(xi, yi; w) is encountered, where\nthe ith term in the sum is the loss of the network prediction on a sample (xi, yi) made by a model with\nparameter w. We assume that the data is partitioned over a total of K clients, with Pk denoting the set of\nindices of the samples on client k, and nk = |Pk|. Thus, the objective in (1) can be re-written as\nf(w) =\nKX\nk=1\nnk\nn Fk(w), where Fk(w) = 1\nnk\nX\ni∈Pk\nfi(w). (2)\nFrontiers 2\n.CC-BY 4.0 International licensemade available under a \n(which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is \nThe copyright holder for this preprintthis version posted May 15, 2024. ; https://doi.org/10.1101/2024.05.14.594167doi: bioRxiv preprint \n\nBishal et al. Sparse FL for NeuroImaging\nIn the typical distributed optimization setting, the IID assumption is made, which says the following: if\nthe partition Pk was created by distributing the training data over the set of clients uniformly at random,\nthen we would have EPk[Fk(w)] = f(w), where the expectation is over the set of examples assigned to a\nfixed client k. In this work, we consider the non-IID setting where this does not hold and Fk could be an\narbitrarily bad approximation to f.\nWhen designing an FL training paradigm, a set of core considerations have to be made to maintain data\nprivacy and address statistical or objective heterogeneity due to the differences in client data and resource\nconstraints at the client sites. A range of work tries to address the issue of heterogeneous non-IID data\nMcMahan et al. (2016); Kulkarni et al. (2020), however, some research also suggests that deterioration in\naccuracy in the FL non-IID setting is almost inevitable Zhao et al. (2018).\n1.3 Federated Learning in Neuro-imaging\nOver the past decade, the field of neuroimaging has strongly embraced data sharing, open-source\nsoftware, and collaboration across multiple sites. This shift is largely attributed to the high costs and time\ndemands associated with neuroimaging data collection Landis et al. (2016); Rootes-Murdy et al. (2022).\nBy pooling data from different sources, researchers can explore findings that extend beyond the initial\nscope of individual studies Poldrack et al. (2013). The practice of sharing data enhances the robustness\nof research through larger sample sizes and the replication of results, offering significant benefits for\nneuroimaging studies. Increasing the sample size not only makes predictions more reliable but also ensures\nthe reliability and validity of research findings, thereby preventing data manipulation and fabrication Ming\net al. (2017); Tenopir et al. (2011). Furthermore, aggregating data can lead to a more diverse sample by\ncombining otherwise similar datasets, thus reflecting a broader range of social health determinants for\nmore comprehensive results Laird (2021). Additionally, reusing data can significantly reduce research costs\nMilham et al. (2018).\nFederated Learning (FL) is increasingly recognized as a transformative approach in healthcare and\nneuro-imaging. In the realm of biomedical imaging, FL has been applied to a variety of tasks. These\ninclude whole-brain segmentation from MRI T1 scans Roy et al. (2019), segmentation of brain tumors\nSheller et al. (2019); Li et al. (2019), multi-site fMRI classification, and the identification of disease\nbiomarkers Li et al. (2020b). COINSTAC Plis et al. (2016a) offers a privacy-focused distributed data\nprocessing framework specifically designed for brain imaging showcasing FL’s role in enhancing privacy\nand efficiency in healthcare data analysis. Additionally, it has been utilized in discovering brain structural\nrelationships across various diseases and clinical cohorts through federated dimensionality reduction from\nshape features Silva et al. (2019).\n1.4 Efficiency in Federated Learning\nThe primary objective of model pruning is to identify subnetworks within larger architectures by\nselectively removing connections. This technique holds considerable appeal for various reasons, particularly\nfor real-time applications on resource-constrained edge devices, which are prevalent in federated\nlearning (FL) and collaborative learning scenarios. Pruning large networks can significantly alleviate\nthe computational demands of inference Elsen et al. (2020) or hardware tailored to exploit sparsity Cerebras\n(2019); Pool et al. (2021). More recently, the lottery ticket hypothesis has emerged Frankle and Carbin\n(2019), suggesting the existence of subnetworks within densely connected networks. These subnetworks,\nwhen trained independently from scratch, can attain comparable accuracy to fully trained dense networks\nFrankle and Carbin (2019), revitalizing the field of sparse deep learning Renda et al. (2020); Chen et al.\nFrontiers 3\n.CC-BY 4.0 International licensemade available under a \n(which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is \nThe copyright holder for this preprintthis version posted May 15, 2024. ; https://doi.org/10.1101/2024.05.14.594167doi: bioRxiv preprint \n\nBishal et al. Sparse FL for NeuroImaging\n(2020). This resurgence of interest has also extended into sparse reinforcement learning (RL) Arnob et al.\n(2021); Sokar et al. (2021). Pruning techniques in deep learning can broadly be categorized into three\ngroups: methods that induce sparsity before training and during initialization Lee et al. (2018); Wang et al.\n(2020); Tanaka et al. (2020), during training Zhu and Gupta (2018); Ma et al. (2019); Yang et al. (2019);\nOhib et al. (2022), and post-training Han et al. (2015); Frankle et al. (2021).\nFor pruning in the FL setting, using a Lottery Ticket like approach would result in immense inefficiency\nin communication. Such methods Frankle and Carbin (2019); Bibikar et al. (2022) usually require costly\npruning and retraining cycles, often training and pruning multiple times to achieve the desired accuracy\nvs sparsity trade-off. Relatively few research have leveraged pruning in the FL paradigm Li et al. (2020a,\n2021); Jiang et al. (2022). In particular, with LotteryFL Li et al. (2020a) and PruneFL Jiang et al. (2022),\nclients need to send the full model to the server regularly resulting in higher bandwidth usage. Moreover,\nin Li et al. (2020a), each client trains a personalized mask to maximize the performance only on the local\ndata. A few recent works Bibikar et al. (2022); Huang et al. (2022); Qiu et al. (2022); Li et al. (2020a)\nalso attempted to leverage sparse training within the FL setting as well. In particular Li et al. (2020a)\nimplemented randomly initialized sparse mask, FedDST Bibikar et al. (2022) built on the idea of RigL Evci\net al. (2020) and mostly focussed on magnitude pruning on the server-side resulting in similar constraints\nand Ohib et al. (2023) uses sparse gradients to efficiently train in a federated learning setting. In this work,\nwe try to alleviate these limitations which we discuss in the following section.\n2 METHOD DESCRIPTION\nIn this section we present our proposed method. We first describe the process of discovering a sub-network\nf(θ ⊙ m) within the full network f(θ), where θ, m ∈ Rd with ∥m∥0 < d . To discover a performant\nsub-network an importance scoring metric is required, which we describe in section 2.1.1. Finally, we\ndelineate our proposed method in section 2.2\n2.1 Sub-network discovery\nGiven a dataset D = {(xi, yi)}n\ni=1 at a site k, the training of a neural network f parameterized by θ ∈ Rd\ncan be written as minimizing the following empirical risk:\nargmin\nθ\n1\nn\nX\ni\nL(f(θ; xi), yi) s.t. θ ∈ H (3)\nwhere θ ∈ Rd and L and H are the loss function and the constraint set respectively.\nIn general, in unconstrained (standard) training the set of possible hypotheses is considered to beH = Rd,\nwhere d is the model dimension. The objective is to minimize the empirical risk L given a training set\n{(xi, yi)}n\ni=1 ∼ D\nat the local client site k. Given access to the gradients of the empirical risk on a batch-\nwise basis, an optimization algorithm such as Stochastic Gradient Descent (SGD) is typically employed to\nachieve the specified objective. This process generates a series of parameter estimates, {θi}T\ni=0\n, where θ0\nrepresents the initial parameters and θT the final optimal parameters. A sub-network within this network is\ndefined as a sparse version of this network with a mask m ∈ {0, 1}|θ| that results in a masked network\nf(θ ⊙ m; xi). When aiming for a target sparsity level where k < d , the parameter pruning challenge\nentails ensuring that the final optimal parameters, θT , have at most k non-zero elements, as denoted by\nthe constraint ∥θT ∥0 ≤ k. In many works, this sparsity constraint applies only to the final parameters and\nFrontiers 4\n.CC-BY 4.0 International licensemade available under a \n(which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is \nThe copyright holder for this preprintthis version posted May 15, 2024. ; https://doi.org/10.1101/2024.05.14.594167doi: bioRxiv preprint \n\nBishal et al. Sparse FL for NeuroImaging\nnot to any intermediate parameter estimates. However, in this work we maintain this sparsity constraint\nthroughout the entire training phase, that is throughout the entire evolution of θ from θ0 to θT .\nThe goal of discovering sub-networks at initialization introduces additional constraints to the previously\ndescribed framework by requiring that all parameter iterations fall within a predetermined subspace of\nH. Specifically, the constraints seek to identify an initial set of parameters, θ0, that has no more than\nk1 non-zero elements ( ∥θ0∥0 ≤ k1), and ensure that all intermediate parameter sets, θi, belong to a\nsubspace ¯H ⊂ H for all i in {1, . . . , T}, where ¯H is the subspace of Rd spanned by the natural basis\nvectors {ej}j∈supp(θ0). Here, supp(θ0) represents the support of θ0, or the set of indices corresponding to\nits non-zero entries. This approach not only specifies a sub-network at initialization with k parameters but\nalso maintains its structure consistently throughout the training.\n2.1.1 Connection Importance Criterion\nLee et al. (2018) introduced a technique inspired by the saliency criterion originally proposed by\nMozer and Smolensky (1988). They contributed an important insight, demonstrating that this criterion is\nremarkably effective in predicting the significance of each connection in a neural network at the initialization\nphase. The core concept revolves around retaining those parameters that, when altered, would have the most\nsubstantial effect on the loss function. This is operationalized by considering a binary vector c ∈ {0, 1}m\nand utilizing the Hadamard product ⊙. Consequently, SNIP calculates the sensitivity of connections based\non this approach. For each connection θj in the network, the importance score for that connection is\ncalculated as following:\ngj(θ; D) = ∂L(θ ⊙ c)\n∂c\n\f\f\n\f\n\f\nc=1\n= ∂L(θ)\n∂θ ⊙ θ (4)\nAfter determining g(θ), the parameters associated with the highestk magnitudes of |gj(θ; D)| are retained.\nEssentially, SNIP prioritizes weights that, regardless of their direction, are distant from the origin and yield\nlarge gradient values. It’s noteworthy that the objective of SNIP can be reformulated as follows De Jorge\net al. (2020); Frankle et al. (2021):\nmax\nc\nS(θ, c) :=\nX\ni∈supp(c)\n|θi∇L(θ)i| s.t. c ∈ {0, 1}m, ∥c∥0 = k. (5)\nIt is trivial to note that the optimal solution to the above problem can be obtained by selecting the indices\ncorresponding to the top-k values of |θi ∇L(θj)|.\n2.2 Proposed Method\nWe propose a novel method for efficient distributed sub-network discovery for distributed neuroimaging\nand propose a method for training such sparse models or subnetworks in a communication efficient\nmanner called Sparse Federated Learning for Neuro-Imaging or NeuroSFL with the goal of tackling\ncommunication inefficiency during decentralized federated learning with non-IID data distribution in the\ncontext of distributed neuro-imaging data. The proposed method initiates with the common initialization\nθ0 at all the local client models. Next, importance scores sj are calculated for each model parameter in the\nnetwork based on the information from the imaging data available across all the client sites. At this stage,\neach client has a unique set of importance scores for their parameters in the local network f based on the\nlocal data available at that site similar to Lee et al. (2018); De Jorge et al. (2020). All the clients transmit\nFrontiers 5\n.CC-BY 4.0 International licensemade available under a \n(which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is \nThe copyright holder for this preprintthis version posted May 15, 2024. ; https://doi.org/10.1101/2024.05.14.594167doi: bioRxiv preprint \n\nBishal et al. Sparse FL for NeuroImaging\nthese scores to each other and a mask m is created corresponding to the top-k % of the aggregated saliency\nscores:\nm = Tk(\nK−1X\nk=0\nsk) (6)\nWhere the Tk is the top-k operator. This mask is then used to train the model fk(θ ⊙ m; x) at site k on\ntheir local data (x, y) ∼ D k.\nFor the federated training among a total of K clients, the clients are trained locally, and at the end of\nlocal training they share their trained parameters which are then averaged; we call this a communication\nround. At the start of this local training, each site k starts with the same initial model weights θ0 which\nat each site k is denoted as θk,0 at training step t = 0 which are then masked with the generated saliency\nmask m to produce the common masked initialization θm\nk,0 as follows:\nθm\nk,0 = θk,0 ⊙ m\nNext these models at each site k are trained on their local dataset (x, y) ∼ D k.\nThe masked models f(θk,0 ⊙ m) across all the sites are trained for a total of T communication rounds to\narrive at the final weights θk,T at each local site. In each communication round t, only a random subset\nF ′ = {f1, f2, ..., fK′} of K′ clients where F ′ ⊆ F the set of all clients, and K′ ≤ K are trained on\ntheir local data (a way to avoid the straggler effect in the real world, wherein a large client group the\nupdate might be bottle-necked by the slowest, most resource-constrained client site). At the end of local\ntraining, the selected subset F ′ of the clients contribute to training the updated model θm\nk,t, which is the\nstarting weights for the next communication round. When sharing the updated weights only the weights\ncorresponding to the 1’s in the binary maskm are shared among the clients and with the server, as only\nthese weights are being trained and the rest of the weights are zero-ed out. This results in the gains in\ncommunication efficiency. To efficiently share the model weights, the clients only share their sparse masked\nweights θm\nF ′ = θF ′ ⊙ m among the selected clients in F ′ using the compressed sparse row (CSR) encoding.\nThe algorithm for the training process is delineated in Algorithm 1.\n2.3 Iterative Importance Score Calculation: IterativeSNIP\nIn this section, we test the effectiveness of iterative-SNIP De Jorge et al. (2020), which is an iterative\nversion of the application of saliency criterion in eq. (4). We briefly describe the iterative-SNIP next. We\nassume k to be the number of parameters to be preserved post pruning. Given that we have some pruning\nschedule (similar to learning rate schedule: linear, exponential etc.) to dividek into a set of natural numbers\n{kt}T\nt=1 such that kt > k t+1 and kT = k. Now, given the binary masking variable ct corresponding to\nkt, the formulation of pruning from kt to kt+1 can be made using the connection sensitivity (4) similar to\nDe Jorge et al. (2020) as:\nct+1 = argmax\nˆθ,c\nS( ¯θ, c) s.t. c ∈ {0, 1}m, ∥c∥0 = kt+1, c ⊙ ct = c, (7)\nwhere ¯θ = θ ⊙ ct. The constraint c ⊙ ct = c is added to ensure that no previously pruning parameter\nis re-activated. Assuming that the pruning schedule ensures a smooth transition from one topology to\nFrontiers 6\n.CC-BY 4.0 International licensemade available under a \n(which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is \nThe copyright holder for this preprintthis version posted May 15, 2024. ; https://doi.org/10.1101/2024.05.14.594167doi: bioRxiv preprint \n\nBishal et al. Sparse FL for NeuroImaging\nanother (∥ct∥0 ≈ ∥ct+1∥0) such that the gradient approximation ∂L( ¯θ)\n∂ ¯θ\n\f\f\n\f\nct\n≈ ∂L( ¯θ)\n∂ ¯θ\n\f\n\f\n\f\nct+1\nis valid, (7) can\nbe approximated as solving (5) at ¯θ.\nIn the scenario where the schedule parameter is set to T = 1 , the original SNIP saliency method is\nrecovered. This is basically employing a gradient approximation approach between the initial dense\nnetwork c0 = 1 and the resulting mask c. We conduct experiments with IterativeSNIP in the federated\nneuroimaging setting and present our findings in section 4.2.\nAlgorithm 1 NeuroSFL\nInput: Total number of clients K; Total communication rounds T\nOutput: Sparse local models ˆθC\nm\n1: Initialize local models with θ0 and transmit to all clients.\n2: m = Tk(PK−1\nk=0 Sk(θ0)) # generate global mask from importance scores\n3: Transmit m to all the sites.\n4: θm\nk,0 ← θk,0 ⊙ m #apply the mask at all sites k = 1, 2, ..., K\n5: for t = 0 to T − 1 do\n6: {ci} ˜K\ni=1 # Sample a set of ˜K clients uniformly from the set of all clients\n7: for site k in parallel for all ˜K clients do\n8: θm\nk,t ← csr(θm\nk,t); #Gather all masked weights θk,m where k ∈ {1, 2, 3, . . . ˜K}\n9: ˆθm\nk,t ←\n\u0010\n1\n˜K\nP ˜K−1\nk=0 θm\nk\n\u0011\n#combine the weights of models in the selected sites\n10: for τ = 0 to N − 1 do\n11: Sample a batch of data δk,t,τ from the local dataset.\n12: gm\nk,t,τ ← ∇ θL( ˆθm\nk,t,τ ; δk,t,τ ) ⊙ m # calculate and mask gradients\n13: ˆθm\nk,t,τ +1 ← ˆθm\nk,t,τ − ηgm\nk,t,τ # take optimization step with masked gradients on masked weights\n14: end for\n15: transmit the non-zero elements of the updated model ˆθm\nk,t,N −1 back to all clients.\n16: end for\n17: end for\n3 EXPERIMENTS\n3.1 Dataset and non-IID partition\nWe evaluated NeuroSFL on the ABCD dataset. ABCD study is the largest long-term study of brain\ndevelopment and child health in the US. It recruited over 10 thousand children of 9 and 10 years old\nfrom 21 sites and followed them for 10 years with annual behavioral and cognitive assessments and\nbiannual MRI scans Garavan et al. (2018). Along with multi-session brain MRI scans for structure and\nfunction, the ABCD study also includes key demographic information including gender, racial information,\nsocio-economic backgrounds, cognitive development, and mental and physical health assessments of the\nsubjects. The ABCD open-source dataset can be found on the National Institute of Mental Health Data\nArchive (NDA) https://nda.nih.gov/. In this study, we used data from the ABCD baseline, which\ncontain 11,875 participants aged 9-10 years.\nT1-weighted MRI images were preprocessed using the Statistical Parametric Mapping 12 (SPM12)\nsoftware toolbox for registration, normalization, and tissue segmentation. Then the gray matter density maps\nFrontiers 7\n.CC-BY 4.0 International licensemade available under a \n(which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is \nThe copyright holder for this preprintthis version posted May 15, 2024. ; https://doi.org/10.1101/2024.05.14.594167doi: bioRxiv preprint \n\nBishal et al. Sparse FL for NeuroImaging\nwere smoothed by a 6 mm3 Gaussian kernel, creating images with the dimensionality of (121,145,121) of\nvoxels at Montreal Neuroimaging Institute (MNI) space with each voxel having dimensions of 1.5×1.5×1.5³\nmm.\nWe simulated the heterogeneous data distributions across federated clients through the adoption of two\ndistinct data partitioning strategies. We outline these strategies for generating non-IID data partitions with\na comprehensive discussion in section 3.1.1.\n3.1.1 Generating non-IID data partition with Dirichlet Distribution\nIn this section, we provide the necessary background on generating non-identical data distribution in the\nclient sites using the Dirichlet Distribution, specifically for the context of federated learning.\n3.1.1.1 non-IID data in FL\nFederated Learning (FL), as introduced by McMahan and Ramage (2017), is a framework designed\nfor training models on decentralized data while preserving privacy. It utilizes the Federated Averaging\n(FedAvg) algorithm where each device, or client, receives a model from a central server, performs stochastic\ngradient descent (SGD) on its local data, and sends the models back for aggregation. Unlike data-center\ntraining where data batches are often IID (independent and identically distributed), FL typically deals with\nnon-IID data distributions across different clients. Hence, to evaluate federated learning it is crucial to not\nmake the IID assumption and instead generate non-IID data among clients for evaluation Hsu et al. (2019).\n3.1.1.2 Generating non-IID data from Dirichlet Distribution\nIn this study, we assume that each client independently chooses training samples. These samples are\nclassified into N distinct classes, with the distribution of class labels governed by a probability vector\nq, which is non-negative and whose components sum to 1, that is, qi > 0, i ∈ [1, N] and ∥q∥1 = 1. For\ngenerating a group of non-identical clients, q ∼ Dir(αp) is drawn from the Dirichlet Distribution, with p\ncharacterizing a prior distribution over the N classes and α controls the degree of identicality among the\nexisting clients and is known as the concentration parameter.\nIn this section, we generate a range of client data partitions from the Dirichlet distribution with a range\nof values for the concentration parameter α for exposition. In fig. 1, we generate a group of 10 balanced\nclients, each holding an equal number of total samples. Similar to Hsu et al. (2019) the prior distribution p\nis assumed to be uniform across all classes. For each client, given a concentration parameter α, we sample\na q from Dir(α) and allocate the corresponding fraction of samples from each client to that client. fig. 1\nillustrates the effect of the concentration parameter α on the class distribution drawn from the Dirichlet\ndistribution on different clients, for the CIFAR-10 dataset. When α → ∞, identical class distribution is\nassigned to each classes. With decreasing α, more non-identicalness is introduced in the class distribution\namong the client population. At the other extreme with α → 0, each class only consists of one particular\nclass.\n3.2 Architecture, Hyperparameters and experimental details\nHere we provide a comprehensive overview of the architecture, hyperparameters, and the experimental\nsetup we use to evaluate our proposed NeuroSFL method on the neuroimaging Adolescent Brain Cognitive\nDevelopment (ABCD) data. Our study focuses on the task of classifying a participant’s sex based on\nMRI scans, by employing a 3D variant of the well-known AlexNet model Krizhevsky et al. (2012). The\nFrontiers 8\n.CC-BY 4.0 International licensemade available under a \n(which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is \nThe copyright holder for this preprintthis version posted May 15, 2024. ; https://doi.org/10.1101/2024.05.14.594167doi: bioRxiv preprint \n\nBishal et al. Sparse FL for NeuroImaging\n(a) (b) (c) (d) (e)\nFigure 1. Generating non-identical client data partitions using the Dirichlet Distribution for the Cifar10\ndataset among 10 clients. Distribution among classes is represented using different colors. (a) Dirichlet,\nα → ∞ results in identical clients (b-d) Client distributions generated from Dirichlet distributions with\ndifferent concentration parameters α (e) Dirichlet, α → 0.0 results in each client being assigned only 1\nparticular class.\n3D variant was referenced from Abrol et al. (2021), which has a specific channel configuration for the\nconvolutional layers set as: 64C-128C-192C-192C-128C, where ’C’ denotes channels.\nWe optimized the learning rate for this task through an exhaustive search ranging from LR=1 × 10−3\nto 1 × 10−6, achieving a delicate balance between rapid convergence and fine-tuning during training. We\nemployed a batch size of 32 and a learning rate decay factor of 0.998 was applied. We applied varying\nsparsity levels, ranging from 0%, 50%, 80%, 90%, and 95% to assess the overall performance. A split of\n80/20 was used for training and testing for each individual site. Our training consists of 5 epochs with 200\ncommunication rounds.\n3.3 Baselines\nWe compared our method with both centralized and decentralized baselines. Centralized baseline includes\nFedAvg McMahan et al. (2017), FedAvg-FT Cheng et al. (2021) which are the standard dense baselines,\nand for the decentralized FL setting, we take the sparse Dis-PFL Dai et al. (2022).\nIn FedAvg McMahan et al. (2017), each client trains its local model using its local data, and then these\nlocal models are aggregated or averaged to update the global model. On the other hand, FedAvg-FT Cheng\net al. (2021) extends the FedAvg algorithm by incorporating fine-tuning or transfer learning. Specifically,\nafter the global model is trained using FedAvg, the global model is then fine-tuned or adapted using\nadditional data from a central server or other external sources. This fine-tuning step allows the global model\nto adapt to new tasks or data distributions beyond what was initially learned from the federated learning\nprocess. We also compare with DisPFL Dai et al. (2022) with varying sparsity levels. DisPFL is a new\nsparse FL technique that randomly prunes each layer similar to Evci et al. (2020) and uses the prune and\nregrow method from that work as well, resulting in a dynamically sparse method.\nIn exploring the impact of using unique local masks instead of a global mask on the performance of\nFL, we established IndividualSNIP as a baseline, representing an approach where unique local masks are\ndevised from the saliency criterion, and local models are trained based on these masks. Moreover, to isolate\nthe impact of just using global masking, that is using the same random mask in all clients, instead of using\ndifferent unique random masks at different sites we compare our method and competing methods against\nrandom global masking as well in fig. 2.\nAdditionally, with further exploration to investigate how different methods of model pruning and selection\nimpact the performance of our approach, we further experiment with other techniques named IterSNIP and\nWeightedSNIP. IterSNIP builds upon the traditional SNIP method Lee et al. (2018) by incorporating multiple\nminibatches during the training process of mask generation. This approach aggregates saliency scores from\nFrontiers 9\n.CC-BY 4.0 International licensemade available under a \n(which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is \nThe copyright holder for this preprintthis version posted May 15, 2024. ; https://doi.org/10.1101/2024.05.14.594167doi: bioRxiv preprint \n\nBishal et al. Sparse FL for NeuroImaging\nthese mini-batches to generate a comprehensive and robust pruning mask. Conversely, WeightedSNIP\nadopts a different strategy, deriving a global mask through a weighted average of saliency scores based on\nthe frequency of data at each site, and assigning importance levels to individual sites based on the amount\nof data at sites.\n4 RESULTS AND DISCUSSION\n4.1 Effect of varying sparsity levels\nWe first explore the effect of sparsity on IID data in section 4.1.1 and then explore the efficacy of\nNeuroSFL on non-IID data in section 4.1.2.\n4.1.1 Effect of varying sparsity levels on IID data\nThe performance of various methods across different sparsity levels was evaluated, as presented in\nTable 1, and visually presented in Figure 2. Sparse baselines, including Ours (NeuroSFL), IndividualSNIP,\nDisPFL Dai et al. (2022), and Global Random Mask, were compared against dense baselines such as\nFedAvg-FT Cheng et al. (2021) and FedAvg McMahan et al. (2017). Notably, our proposed NeuroSFL,\nexhibited robust performance across varying sparsity levels, achieving an accuracy of 92.52% at 0% sparsity\nand maintaining high accuracy even at higher sparsity levels, with 71.18% accuracy at 95% sparsity. In\ncomparison, IndividualSNIP demonstrated decreasing accuracy as sparsity increased, with a significant\ndrop to 52.70% at 95% sparsity. This is in line with expectation as individual-SNIP only incorporates the\nsaliency scores from a single site at random and does not incorporate information from the datasets at all\nthe participating cites.\nMoreover, in contrast to NeuroSFL, DisPFL, and Global Random Mask also showcased diminishing\naccuracy with increasing sparsity, highlighting the effectiveness of our proposed approach in mitigating the\nadverse effects of sparsity on model performance on neuroimaging data. Notably, Global Random Mask\noutperformed DisPFL on lower sparsities, suggesting that in general global random masks might be more\nsuitable for federated applications compared to even targeted unique local masks which DisPFL employs.\nDense baselines, such as FedAvg-FT and FedAvg, even while being not sparse and using full\ncommunication achieved comparable performances to NeuroSFL in the non-extreme sparsity region.\nFigure 2. (a) Comparison of methods for gender classification using MRI Scans of ABCD dataset. (b)\nGender differences in each of the 21 ABCD sites along with the performance of the global model on each\nsite with 50% sparsity.\nFrontiers 10\n.CC-BY 4.0 International licensemade available under a \n(which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is \nThe copyright holder for this preprintthis version posted May 15, 2024. ; https://doi.org/10.1101/2024.05.14.594167doi: bioRxiv preprint \n\nBishal et al. Sparse FL for NeuroImaging\nMethod\nSparsity (%)\n0%\n50% 80% 90% 95%\nSparse\nBaselines\nOurs\n(NeuroSFL) 92.52% 92.4% 88.19% 87.5% 71.18%\nDisPFL 85.24% 79.78% 76.00% 74.01% 62.12%\nIndividualSNIP 91.59% 80.20% 52.37% 51.04% 52.70%\nGlobal Random Mask 90.39% 89.20% 84.48% 71.44% 47.53%\nDense\nBaselines\nFedA\nvg-FT 92.1% (Dense Baseline)\nFedAvg 90.5% (Dense Baseline)\nTable 1. Performance comparison of different methods and sparsity levels. Sparse baselines include Ours-\n(NeuroSFL), DistPFL, IndividualSNIP, and Global Random Mask. Dense baselines include FedAvg and\nFedAvg-FT.\nNeuroSFL even surpassed the performance of dense baselines at a sparsity level of 50%, highlighting\nthe effectiveness of our proposed sparse method in optimizing model performance while reducing\ncommunication costs. Furthermore, Figure 2 (b) illustrates that the single global model trained with\nNeuroSFL demonstrated excellent performance for data within each site, emphasizing the model’s\neffectiveness in capturing site-specific characteristics while maintaining high accuracy.\nAdditionally, in Figure 3 (a), it is observed that the performance of local models trained with NeuroSFL\nremains consistently robust across non-IID states of local data, indicating the model’s versatility and\nreliability in various data distribution scenarios.\n4.1.2 Effect of varying sparsity levels on non-IID data:\nExploring the Impact of Varied Sparsity Levels on Non-IID Data for 10 Clients:\nIn this subsection, we delve into the influence of changing sparsity levels on our model’s performance on\nnon-IID data for 10 clients, employing the Dirichlet Distribution with alpha = 0.3 across various sparsity\nlevels. Fig. 4 provides a visual representation of the accuracy versus sparsity relationship, showcasing\nthe consistent accuracy achieved across different sparsity levels (Figure 4 (a)). Additionally, Figure 4 (b)\nillustrates the class distribution with Dir(0.3) for the ABCD dataset for 10 clients and their final local test\n(a) (b)\nFigure 3. (a) Gender differences in each of the 21 ABCD sites along with the performance of the local\nmodels on each site with 50% sparsity. (b) Sample structural MRI brain image of brains used in gender\nclassification.\nFrontiers 11\n.CC-BY 4.0 International licensemade available under a \n(which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is \nThe copyright holder for this preprintthis version posted May 15, 2024. ; https://doi.org/10.1101/2024.05.14.594167doi: bioRxiv preprint \n\nBishal et al. Sparse FL for NeuroImaging\nFigure 4. (a) Comparison of methods for gender classification using non-IID Dirichlet distribution with\nalpha=0.3 and varying sparsity levels. (b) Gender differences in each of the 10 ABCD sites along with the\nperformance of the model.\naccuracy. Following the Dirichlet partition, we have uneven data distribution, e.g. with site-6 having a total\n5459 samples and sites-3 and 9 having only 8 and 5 training samples respectively.\nNotably, our model demonstrated robust accuracy across different sparsity levels, ranging from 89.20%\nto 93.19%. These results underscore the resilience of our approach in maintaining high performance\neven under significant sparsity constraints. Importantly, when examining the local test accuracy across\ndifferent clients, we observed consistently good performance regardless of the data differences. This\nresilience suggests that our model’s effectiveness extends beyond homogeneous datasets, making it suitable\nfor deployment in federated learning scenarios with diverse client characteristics, specifically non-IID\ndistribution.\nExploring the Impact of Varied Sparsity Levels on Non-IID Data for 30 Clients: We extend our\nanalysis to incorporate 30 clients and explore the influence of changing sparsity levels on our model’s\nperformance on non-IID data. We continue to employ the Dirichlet Distribution with alpha = 0.3 across\nvarious sparsity levels. Fig. 5 provides a visual representation of the accuracy versus sparsity relationship,\nshowcasing the consistent accuracy achieved across different sparsity levels (fig. 5 (a)). Additionally, fig. 5\n(b) illustrates the class distribution with Dir(0.3) for the ABCD dataset for 30 clients and their final local\ntest accuracy. The results reveal a similar trend to the 10-client scenario, with the model achieving notable\naccuracy across varying sparsity levels. However, for the highly sparse condition of 95%, there is a slight\ndrop in accuracy to 63.75%. This decline can be attributed to the increased difficulty for the model to\ngeneralize with such sparse data under the additional restriction of having larger clients. Despite this\nchallenge, our model maintains its effectiveness across a diverse range of sparsity levels, indicating its\npotential for practical applications in federated learning scenarios with larger amount of client sites.\n4.2 IterativeSNIP Performance\nWe evaluate the performance of IterSNIP and WeightedSNIP to explore their efficacy in sparse FL\nscenarios. Table 2 summarizes the accuracy results obtained at 50% sparsity for different iterations of\nIterSNIP and WeightedSNIP. IterSNIP, with varying numbers of iterations (1, 10, and 20), demonstrated\nconsistent performance with increasing iterations, achieving accuracies of 92.4%, 91.82%, and 92.67%,\nrespectively. These results suggest that utilizing multiple iterations to obtain SNIP masks does not\nnecessarily enhance model performance in scenarios with sparsity constraints, and especially when\nused on neuroimaging data. This is a departure from the single-node case on natural image datasets such as\nFrontiers 12\n.CC-BY 4.0 International licensemade available under a \n(which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is \nThe copyright holder for this preprintthis version posted May 15, 2024. ; https://doi.org/10.1101/2024.05.14.594167doi: bioRxiv preprint \n\nBishal et al. Sparse FL for NeuroImaging\nFigure 5. (a) Comparison of methods for gender classification using non-IID dirichlet distribution with\nalpha=0.3 and varying sparsity levels. (b) Gender differences in each of the 30 ABCD sites along with the\nperformance of the model for 50% sps.\nCIFAR10 or CIFAR100 De Jorge et al. (2020). Similarly, WeightedSNIP, which incorporates weighted\naverages of saliency scores, achieved an accuracy of 92.10% and does not outperform the vanilla averaging\ntechnique. This proves that our model is robust enough to find a sparse mask, with minimal effect from the\namount of data at each sites.\nMethod Iterations Accuracy (50% Sparsity)\nIterSNIP\n1 Iteration 92.40%\n10 Iteration 91.82%\n20 Iteration 92.67%\nWeightedSNIP 1 Iteration 92.10%\nTable 2. Performance comparison of IterSNIP with different iterations and WeightedSNIP in terms of\naccuracy at sparsity of 50%.\n4.3 Wall-time efficiency gains in the real world COINSTAC system\nIn our pursuit of optimizing federated learning methodologies, we make use of Coinstac Plis et al. (2016b),\na cutting-edge open-source federated learning solution designed for collaborative neuroimaging endeavors\nat scale. Deployed in real-world scenarios, Coinstac embodies a paradigm shift in collaborative research,\ntranscending traditional boundaries and fostering synergistic interactions among researchers worldwide.\nCoinstac’s architecture facilitates decentralized computations across a distributed network of geographically\ndispersed client nodes, seamlessly integrating diverse computational tasks while safeguarding data privacy\nthrough state-of-the-art differential privacy mechanisms. Having said this, our experiment leverages\nCoinstac’s robust infrastructure to benchmark our method against the standard dense FedAvg algorithm\nMcMahan et al. (2017) within a practical real-world context. Our evaluation spans five diverse client\nlocations, spanning from North Virginia to Frankfurt, each representing a distinct geographical node within\nCoinstac’s decentralized network. By meticulously assessing the mean communication time—reflecting\nthe duration for the server model to aggregate all client weights during each communication round, we\ndemonstrate the efficiency of our algorithm in optimizing federated learning workflows.\nFrontiers 13\n.CC-BY 4.0 International licensemade available under a \n(which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is \nThe copyright holder for this preprintthis version posted May 15, 2024. ; https://doi.org/10.1101/2024.05.14.594167doi: bioRxiv preprint \n\nBishal et al. Sparse FL for NeuroImaging\nOur investigation encompasses five local client models, each featuring varying sizes or depths of ResNet\narchitectures while maintaining a sparsity level of 90% across experiments. The ensuing comparative\nanalysis, delineated in table 3, demonstrates the tangible speed enhancements achieved by our proposed\nmethodology as compared to the standard FedAvg. Importantly, our results indicate that our approach\nconsistently outperforms FedAvg across all ResNet architectures. For instance, in the case of ResNet32,\nour method achieves a communication time of 0.238 ± 0.02 seconds, compared to 0.285 ± 0.04 seconds\nfor FedAvg, resulting in a speedup of 1.20x. This trend continues across deeper architectures, with\nour technique demonstrating significant improvements in communication efficiency. For instance, for\nResNet110, our method achieves a remarkable speedup of 2.32x over FedAvg, showcasing its ability to\nhandle complex models with greater efficiency. These empirical findings underscore the importance of\nsparse federated techniques like NeuroSFL, thereby propelling collaborative neuroimaging research to\nunprecedented heights.\nAr\nchitecture Accuracy Communication\nTime (s) Speed\nup\nFedA\nvg NeuroSFL\nResNet32 90.52% 0.285 ± 0.04 0.238 ± 0.02 1.20x\nResNet44 89.65% 0.409 ± 0.06 0.328 ± 0.04 1.24x\nResNet56 93.74% 0.531 ± 0.07 0.407 ± 0.06 1.30x\nResNet110 93.25% 1.812 ± 0.33 0.781 ± 0.13 2.32x\nTable 3. Comparison of communication time between FedAvg and NeuroSFL on Cifar10 for ResNet\narchitectures of different depth.\n4.4 Sparsity vs Accuracy Performance comparison\nIn this section we analyze and interpret the results from section 4. First, we probe the reasons behind the\nperformance gains in comparison to a state of the art federated sparse learning method Dai et al. (2022).\nIn a specific comparison with DistPFL, we can see that NeuroSFL consistently performs better than\nDisPFL in a range of sparsities in the selected tasks. This is probably due to a better choice of the initial\nsparse sub-network using the importance criterion. Another difference is that, in DisPFL different local\nclients have different levels of sparsity and a final model averaging is done, where the final model becomes\ndenser due to the union of many sparse subnetworks. We however retain the same mask in all the clients\nand start from the same initialization in all the clients, result in equivalent sparsity in all the clients; this\nalso leaves open the potential of keeping sparse global models in a centralized FL setting.\n5 CONCLUSION\nIn this paper, we elaborate a novel sparse decentralized federated learning approach that trains a sparse\nmodel efficiently for collaborative training on neuro-imaging data. By extending a gradient-based parameter\nimportance criterion to the FL setting, we achieve reduced communication costs and better bandwidth\nin decentralized training. Our method leverages the nature of local data distribution, resulting in a client\ndata-aware global sparse mask. This leads to savings in communication time and bandwidth during sparse\ntraining. We test our approach on the ABCD dataset and report improved performance compared to\ncontemporary methods. Overall, our sparse FL technique enhances communication time, making it suitable\nfor bandwidth-limited settings without compromising accuracy.\nFrontiers 14\n.CC-BY 4.0 International licensemade available under a \n(which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is \nThe copyright holder for this preprintthis version posted May 15, 2024. ; https://doi.org/10.1101/2024.05.14.594167doi: bioRxiv preprint \n\nBishal et al. Sparse FL for NeuroImaging\nAUTHOR CONTRIBUTIONS\nBT and RO contributed equally in this work and share first authorship. EG assisted in manuscript preparation\nand helped in ideation. JL, VC and SP acted as advisors for the project and assisted in manuscript preparation\nand funded the project.\nFUNDING\nThis work was supported by R01DA040487 and R01DA049238 from NIH and 2112455 from NSF.\nREFERENCES\nAbrol, A., Fu, Z., Salman, M., Silva, R., Du, Y ., Plis, S., et al. (2021). Deep learning encodes\nrobust discriminative neuroimaging representations to outperform standard machine learning. Nature\nCommunications 12. doi:10.1038/s41467-020-20655-6\nArnob, S. Y ., Ohib, R., Plis, S., and Precup, D. (2021). Single-shot pruning for offline reinforcement\nlearning. arXiv preprint arXiv:2112.15579\nBibikar, S., Vikalo, H., Wang, Z., and Chen, X. (2022). Federated dynamic sparse training: Computing\nless, communicating less, yet learning better. In Proceedings of the AAAI Conference on Artificial\nIntelligence. vol. 36, 6080–6088\nBonawitz, K., Eichner, H., Grieskamp, W., Huba, D., Ingerman, A., Ivanov, V ., et al. (2019). Towards\nfederated learning at scale: System design. Proceedings of machine learning and systems 1, 374–388\n[Dataset] Cerebras (2019). Wafer Scale Engine: Why We Need Big Chips for Deep\nLearning. https://cerebras.net/blog/cerebras-wafer-scale-engine-why-we-\nneed-big-chips-for-deep-learning/\nChen, T., Frankle, J., Chang, S., Liu, S., Zhang, Y ., Wang, Z., et al. (2020). The lottery ticket hypothesis\nfor pre-trained bert networks. Advances in neural information processing systems 33, 15834–15846\nCheng, G., Chadha, K., and Duchi, J. (2021). Fine-tuning is fine in federated learning. arXiv preprint\narXiv:2108.07313\nDai, R., Shen, L., He, F., Tian, X., and Tao, D. (2022). Dispfl: Towards communication-efficient\npersonalized federated learning via decentralized sparse training. arXiv preprint arXiv:2206.00187\nDe Jorge, P., Sanyal, A., Behl, H. S., Torr, P. H., Rogez, G., and Dokania, P. K. (2020). Progressive\nskeletonization: Trimming more fat from a network at initialization. arXiv preprint arXiv:2006.09081\nElsen, E., Dukhan, M., Gale, T., and Simonyan, K. (2020). Fast sparse convnets. In Proceedings of the\nIEEE/CVF conference on computer vision and pattern recognition. 14629–14638\nEvci, U., Gale, T., Menick, J., Castro, P. S., and Elsen, E. (2020). Rigging the lottery: Making all tickets\nwinners. In International Conference on Machine Learning (PMLR), 2943–2952\nFrankle, J. and Carbin, M. (2019). The lottery ticket hypothesis: Finding sparse, trainable neural networks.\nIn 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May\n6-9, 2019\nFrankle, J., Dziugaite, G. K., Roy, D., and Carbin, M. (2021). Pruning neural networks at initialization:\nWhy are we missing the mark? In 9th International Conference on Learning Representations, ICLR\n2021, Virtual Event, Austria, May 3-7, 2021\nGaravan, H., Bartsch, H., Conway, K., Decastro, A., Goldstein, R., Heeringa, S., et al. (2018). Recruiting\nthe abcd sample: Design considerations and procedures. Developmental cognitive neuroscience 32,\n16–22\nFrontiers 15\n.CC-BY 4.0 International licensemade available under a \n(which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is \nThe copyright holder for this preprintthis version posted May 15, 2024. ; https://doi.org/10.1101/2024.05.14.594167doi: bioRxiv preprint \n\nBishal et al. Sparse FL for NeuroImaging\nHan, S., Mao, H., and Dally, W. J. (2015). Deep compression: Compressing deep neural networks with\npruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149\nHsu, T.-M. H., Qi, H., and Brown, M. (2019). Measuring the effects of non-identical data distribution for\nfederated visual classification. arXiv preprint arXiv:1909.06335\nHuang, T., Liu, S., Shen, L., He, F., Lin, W., and Tao, D. (2022). Achieving personalized federated learning\nwith sparse local models. arXiv preprint arXiv:2201.11380\nJiang, Y ., Wang, S., Valls, V ., Ko, B. J., Lee, W.-H., Leung, K. K., et al. (2022). Model pruning enables\nefficient federated learning on edge devices. IEEE Transactions on Neural Networks and Learning\nSystems\nKoneˇcn`y, J., McMahan, H. B., Ramage, D., and Richt´arik, P. (2016). Federated optimization: Distributed\nmachine learning for on-device intelligence. arXiv preprint arXiv:1610.02527\nKrizhevsky, A., Sutskever, I., and Hinton, G. E. (2012). Imagenet classification with deep convolutional\nneural networks. In Advances in Neural Information Processing Systems, eds. F. Pereira, C. Burges,\nL. Bottou, and K. Weinberger (Curran Associates, Inc.), vol. 25\nKulkarni, V ., Kulkarni, M., and Pant, A. (2020). Survey of personalization techniques for federated\nlearning. In 2020 Fourth World Conference on Smart Trends in Systems, Security and Sustainability\n(WorldS4) (IEEE), 794–797\nLaird, A. R. (2021). Large, open datasets for human connectomics research: Considerations for reproducible\nand responsible data use. NeuroImage 244, 118579\nLandis, D., Courtney, W., Dieringer, C., Kelly, R., King, M., Miller, B., et al. (2016). Coins data exchange:\nAn open platform for compiling, curating, and disseminating neuroimaging data. NeuroImage 124,\n1084–1088\nLee, N., Ajanthan, T., and Torr, P. H. (2018). Snip: Single-shot network pruning based on connection\nsensitivity. arXiv preprint arXiv:1810.02340\nLi, A., Sun, J., Wang, B., Duan, L., Li, S., Chen, Y ., et al. (2020a). Lotteryfl: Personalized and\ncommunication-efficient federated learning with lottery ticket hypothesis on non-iid datasets. arXiv\npreprint arXiv:2008.03371\nLi, A., Sun, J., Zeng, X., Zhang, M., Li, H., and Chen, Y . (2021). Fedmask: Joint computation and\ncommunication-efficient personalized federated learning via heterogeneous masking. In Proceedings of\nthe 19th ACM Conference on Embedded Networked Sensor Systems. 42–55\nLi, W., Milletar`ı, F., Xu, D., Rieke, N., Hancox, J., Zhu, W., et al. (2019). Privacy-preserving federated\nbrain tumour segmentation. In Machine Learning in Medical Imaging: 10th International Workshop,\nMLMI 2019, Held in Conjunction with MICCAI 2019, Shenzhen, China, October 13, 2019, Proceedings\n10 (Springer), 133–141\nLi, X., Gu, Y ., Dvornek, N., Staib, L. H., Ventola, P., and Duncan, J. S. (2020b). Multi-site fmri analysis\nusing privacy-preserving federated learning and domain adaptation: Abide results. Medical Image\nAnalysis 65, 101765\nMa, R., Miao, J., Niu, L., and Zhang, P. (2019). Transformed ℓ1 regularization for learning sparse deep\nneural networks. Neural Networks 119, 286–298\nMcMahan, B., Moore, E., Ramage, D., Hampson, S., and y Arcas, B. A. (2017). Communication-efficient\nlearning of deep networks from decentralized data. In Artificial intelligence and statistics (PMLR),\n1273–1282\n[Dataset] McMahan, B. and Ramage, D. (2017). Federated learning: Collaborative machine learning\nwithout centralized training data\nFrontiers 16\n.CC-BY 4.0 International licensemade available under a \n(which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is \nThe copyright holder for this preprintthis version posted May 15, 2024. ; https://doi.org/10.1101/2024.05.14.594167doi: bioRxiv preprint \n\nBishal et al. Sparse FL for NeuroImaging\nMcMahan, H. B., Moore, E., Ramage, D., Hampson, S., and y Arcas, B. A. (2016). Communication-\nefficient learning of deep networks from decentralized data. In International Conference on Artificial\nIntelligence and Statistics\nMilham, M. P., Craddock, R. C., Son, J. J., Fleischmann, M., Clucas, J., Xu, H., et al. (2018). Assessment\nof the impact of shared brain imaging data on the scientific literature. Nature Communications 9, 2818\nMing, J., Verner, E., Sarwate, A., Kelly, R., Reed, C., Kahleck, T., et al. (2017). Coinstac: Decentralizing\nthe future of brain imaging analysis. F1000Research 6\nMozer, M. C. and Smolensky, P. (1988). Skeletonization: A technique for trimming the fat from a network\nvia relevance assessment. Advances in neural information processing systems 1\nOhib, R., Gillis, N., Dalmasso, N., Shah, S., Potluru, V . K., and Plis, S. (2022). Explicit group sparse\nprojection with applications to deep learning and NMF. Transactions on Machine Learning Research\nOhib, R., Thapaliya, B., Gaggenapalli, P., Liu, J., Calhoun, V ., and Plis, S. (2023). Salientgrads: Sparse\nmodels for communication efficient and data aware distributed federated training. arXiv preprint\narXiv:2304.07488\nPlis, S. M., Sarwate, A. D., Wood, D., Dieringer, C., Landis, D., Reed, C., et al. (2016a). Coinstac: a\nprivacy enabled model and prototype for leveraging and processing decentralized brain imaging data.\nFrontiers in neuroscience10, 204805\nPlis, S. M., Sarwate, A. D., Wood, D., Dieringer, C., Landis, D., Reed, C., et al. (2016b). COINSTAC: A\nprivacy enabled model and prototype for leveraging and processing decentralized brain imaging data.\nFrontiers in Neuroscience10. doi:10.3389/fnins.2016.00365\nPoldrack, R. A., Barch, D. M., Mitchell, J. P., Wager, T. D., Wagner, A. D., Devlin, J. T., et al. (2013).\nToward open sharing of task-based fmri data: the openfmri project. Frontiers in neuroinformatics7, 12\n[Dataset] Pool, J., Sawarkar, A., and Rodge, J. (2021). Accelerating Inference with\nSparsity Using the NVIDIA Ampere Architecture and NVIDIA TensorRT. https:\n//developer.nvidia.com/blog/accelerating-inference-with-sparsity-\nusing-ampere-and-tensorrt/\nQiu, X., Fernandez-Marques, J., Gusmao, P. P., Gao, Y ., Parcollet, T., and Lane, N. D. (2022). Zerofl:\nEfficient on-device training for federated learning with local sparsity. arXiv preprint arXiv:2208.02507\nRenda, A., Frankle, J., and Carbin, M. (2020). Comparing rewinding and fine-tuning in neural network\npruning. arXiv preprint arXiv:2003.02389\nRootes-Murdy, K., Gazula, H., Verner, E., Kelly, R., DeRamus, T., Plis, S., et al. (2022). Federated analysis\nof neuroimaging data: a review of the field. Neuroinformatics , 1–14\nRoy, A. G., Siddiqui, S., P¨olsterl, S., Navab, N., and Wachinger, C. (2019). Braintorrent: A peer-to-peer\nenvironment for decentralized federated learning. arXiv preprint arXiv:1905.06731\nSheller, M. J., Reina, G. A., Edwards, B., Martin, J., and Bakas, S. (2019). Multi-institutional deep learning\nmodeling without sharing patient data: A feasibility study on brain tumor segmentation. In Brainlesion:\nGlioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries: 4th International Workshop, BrainLes\n2018, Held in Conjunction with MICCAI 2018, Granada, Spain, September 16, 2018, Revised Selected\nPapers, Part I 4(Springer), 92–104\nSilva, S., Gutman, B. A., Romero, E., Thompson, P. M., Altmann, A., and Lorenzi, M. (2019). Federated\nlearning in distributed medical databases: Meta-analysis of large-scale subcortical brain data. In 2019\nIEEE 16th international symposium on biomedical imaging (ISBI 2019) (IEEE), 270–274\nSokar, G., Mocanu, E., Mocanu, D. C., Pechenizkiy, M., and Stone, P. (2021). Dynamic sparse training for\ndeep reinforcement learning. arXiv preprint arXiv:2106.04217\nFrontiers 17\n.CC-BY 4.0 International licensemade available under a \n(which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is \nThe copyright holder for this preprintthis version posted May 15, 2024. ; https://doi.org/10.1101/2024.05.14.594167doi: bioRxiv preprint \n\nBishal et al. Sparse FL for NeuroImaging\nTanaka, H., Kunin, D., Yamins, D. L., and Ganguli, S. (2020). Pruning neural networks without any data\nby iteratively conserving synaptic flow. In Advances in Neural Information Processing Systems 33:\nAnnual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12,\n2020, virtual, eds. H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin\nTenopir, C., Allard, S., Douglass, K., Aydinoglu, A. U., Wu, L., Read, E., et al. (2011). Data sharing by\nscientists: practices and perceptions. PloS one 6, e21101\nWang, C., Zhang, G., and Grosse, R. B. (2020). Picking winning tickets before training by preserving\ngradient flow. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa,\nEthiopia, April 26-30, 2020\nYang, H., Wen, W., and Li, H. (2019). Deephoyer: Learning sparser neural network with differentiable\nscale-invariant sparsity measures. arXiv preprint arXiv:1908.09979\nZhao, Y ., Li, M., Lai, L., Suda, N., Civin, D., and Chandra, V . (2018). Federated learning with non-iid data.\narXiv preprint arXiv:1806.00582\nZhu, M. and Gupta, S. (2018). To prune, or not to prune: Exploring the efficacy of pruning for model\ncompression. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC,\nCanada, April 30 - May 3, 2018, Workshop Track Proceedings\nFrontiers 18\n.CC-BY 4.0 International licensemade available under a \n(which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is \nThe copyright holder for this preprintthis version posted May 15, 2024. ; https://doi.org/10.1101/2024.05.14.594167doi: bioRxiv preprint","source_license":"CC-BY-4.0","license_restricted":false}