专属领域论文订阅
“ VX关注晓理紫，每日更新论文，如感兴趣，请转发给有需要的同学，谢谢支持
“ 如果你感觉对你有所帮助，请关注我，每日准时为你推送最新论文。
“ 为了答谢各位网友的支持，从今日起免费为300名读者提供订阅主题论文服务，只需关注公号并在留言中提供{邮箱+论文主题}（如：123456@xx.com + chatgpt@large language model @LLM）,主题必须是同一个领域，最多三个关键词。解释权归博主所有
“ 分类:大语言模型LLM
视觉模型VLM
扩散模型
视觉语言导航VLN
强化学习 RL
模仿学习 IL
机器人
开放词汇，检测分割
== RL ==
标题: Dual RL: Unification and New Methods for Reinforcement and Imitation Learning
作者: Harshit Sikchi, Qinqing Zheng, Amy Zhang
PubTime: 2024-01-26
Downlink: http://arxiv.org/abs/2302.08560v3
Project: https://hari-sikchi.github.io/dual-rl|
中文摘要: 强化学习（RL）的目标是找到一个最大化预期累积回报的策略。已经表明，这个目标可以表示为线性约束下状态——行动访问分配的优化问题。这个公式的对偶问题，我们称之为对偶RL，是无约束的，更容易优化。在这项工作中，我们首先将几个最先进的离线RL和离线模仿学习（IL）算法作为具有共享结构的双重RL方法的实例。这种统一使我们能够确定现有方法缺点的根本原因。对于离线IL，我们的分析表明，现有的方法是基于限制性覆盖假设，这极大地限制了它们在实践中的性能。为了解决这一限制，我们提出了一种新的无鉴别器方法反冲，该方法从任意非策略数据中学习模仿，以获得接近专家的性能。对于离线RL，我们的分析将最近的离线RL方法XQL框架在对偶框架中，我们进一步提出了一种新的方法f-DVL，它为Gumbel回归损失提供了替代选择，修复了XQL的已知训练不稳定性问题。我们提出的两种方法反冲和f-DVL在IL和RL中的性能改进在大量模拟机器人运动和操纵任务中得到验证。项目代码和细节可以在这个https：//hari-sikchi.github.io/dual-rl。
摘要: The goal of reinforcement learning (RL) is to find a policy that maximizes the expected cumulative return. It has been shown that this objective can be represented as an optimization problem of state-action visitation distribution under linear constraints. The dual problem of this formulation, which we refer to as dual RL, is unconstrained and easier to optimize. In this work, we first cast several state-of-the-art offline RL and offline imitation learning (IL) algorithms as instances of dual RL approaches with shared structures. Such unification allows us to identify the root cause of the shortcomings of prior methods. For offline IL, our analysis shows that prior methods are based on a restrictive coverage assumption that greatly limits their performance in practice. To fix this limitation, we propose a new discriminator-free method ReCOIL that learns to imitate from arbitrary off-policy data to obtain near-expert performance. For offline RL, our analysis frames a recent offline RL method XQL in the dual framework, and we further propose a new method f-DVL that provides alternative choices to the Gumbel regression loss that fixes the known training instability issue of XQL. The performance improvements by both of our proposed methods, ReCOIL and f-DVL, in IL and RL are validated on an extensive suite of simulated robot locomotion and manipulation tasks. Project code and details can be found at this https://hari-sikchi.github.io/dual-rl.
标题: Decoupled Prioritized Resampling for Offline RL
作者: Yang Yue, Bingyi Kang, Xiao Ma
PubTime: 2024-01-26
Downlink: http://arxiv.org/abs/2306.05412v3
GitHub: https://github.com/sail-sg/OPER|
中文摘要: 离线强化学习（RL）受到分布移位问题的挑战。为了解决这个问题，现有的工作主要集中在设计学习策略和行为策略之间复杂的策略约束。然而，通过统一抽样，这些约束同样适用于表现良好和表现不佳的行为，这可能会对学习到的策略产生负面影响。为了缓解这个问题，我们提出了离线优先体验回放（OPER），具有一类优先功能，旨在优先考虑高回报的过渡，使它们在培训期间更频繁地被访问。通过理论分析，我们表明这类优先级函数诱导了一种改进的行为策略，并且当受限于这种改进的策略时，策略约束的离线RL算法可能产生更好的解决方案。我们开发了两种实用的策略，通过基于拟合值网络（OPER-A）估计优势或利用轨迹回报（OPER-R）进行快速计算来获得优先级权重。OPER是离线RL算法的即插即用组件。作为案例研究，我们在五种不同的算法上评估了OPER，包括BC、TD3+BC、Onestep RL、CQL和IQL。大量实验表明，OPER-A和OPER-R都显著提高了所有基线方法的性能。代码和优先权重可在https://github.com/sail-sg/OPER。
摘要: Offline reinforcement learning (RL) is challenged by the distributional shift problem. To address this problem, existing works mainly focus on designing sophisticated policy constraints between the learned policy and the behavior policy. However, these constraints are applied equally to well-performing and inferior actions through uniform sampling, which might negatively affect the learned policy. To alleviate this issue, we propose Offline Prioritized Experience Replay (OPER), featuring a class of priority functions designed to prioritize highly-rewarding transitions, making them more frequently visited during training. Through theoretical analysis, we show that this class of priority functions induce an improved behavior policy, and when constrained to this improved policy, a policy-constrained offline RL algorithm is likely to yield a better solution. We develop two practical strategies to obtain priority weights by estimating advantages based on a fitted value network (OPER-A) or utilizing trajectory returns (OPER-R) for quick computation. OPER is a plug-and-play component for offline RL algorithms. As case studies, we evaluate OPER on five different algorithms, including BC, TD3+BC, Onestep RL, CQL, and IQL. Extensive experiments demonstrate that both OPER-A and OPER-R significantly improve the performance for all baseline methods. Codes and priority weights are availiable at https://github.com/sail-sg/OPER.
标题: Off-Policy Primal-Dual Safe Reinforcement Learning
作者: Zifan Wu, Bo Tang, Qian Lin
PubTime: 2024-01-26
Downlink: http://arxiv.org/abs/2401.14758v1
GitHub: https://github.com/ZifanWu/CAL|
中文摘要: 原始对偶安全RL方法通常在策略的原始更新和拉格朗日乘数的对偶更新之间执行迭代。这种训练范例非常容易受到累积成本估计中的误差的影响，因为该估计充当连接原始和双重更新过程的关键纽带。我们表明，当使用非策略方法时，这个问题会导致成本的显著低估，从而导致无法满足安全约束。为了解决这个问题，我们提出了\textit{保守策略优化}，它通过考虑成本估计中的不确定性来学习约束满足区域中的策略。这提高了约束满意度，但也潜在地阻碍了奖励最大化。然后，我们引入\textit{局部策略凸化}，通过逐渐减少估计不确定性来帮助消除这种次优性。我们提供了这两种成分的联合耦合效应的理论解释，并通过大量的实验进一步验证了它们。基准任务的结果表明，我们的方法不仅在使用更少样本的情况下实现了与最先进的策略方法相当的渐近性能，而且显著减少了训练期间的约束违反。我们的代码可在https：//github.com/ZifanWu/CAL。获得
摘要: Primal-dual safe RL methods commonly perform iterations between the primal update of the policy and the dual update of the Lagrange Multiplier. Such a training paradigm is highly susceptible to the error in cumulative cost estimation since this estimation serves as the key bond connecting the primal and dual update processes. We show that this problem causes significant underestimation of cost when using off-policy methods, leading to the failure to satisfy the safety constraint. To address this issue, we propose \textit{conservative policy optimization}, which learns a policy in a constraint-satisfying area by considering the uncertainty in cost estimation. This improves constraint satisfaction but also potentially hinders reward maximization. We then introduce \textit{local policy convexification} to help eliminate such suboptimality by gradually reducing the estimation uncertainty. We provide theoretical interpretations of the joint coupling effect of these two ingredients and further verify them by extensive experiments. Results on benchmark tasks show that our method not only achieves an asymptotic performance comparable to state-of-the-art on-policy methods while using much fewer samples, but also significantly reduces constraint violation during training. Our code is available at https://github.com/ZifanWu/CAL.
标题: Beyond Task Performance: Evaluating and Reducing the Flaws of Large Multimodal Models with In-Context Learning
作者: Mustafa Shukor, Alexandre Rame, Corentin Dancette
PubTime: 2024-01-22
Downlink: http://arxiv.org/abs/2310.00647v2
Project: https://evalign-icl.github.io/|
GitHub: https://github.com/mshukor/EvALign-ICL|
中文摘要: 随着大型语言模型（LLMs）的成功，大型多模态模型（LMMs），如Flamingo模型及其后续竞争对手，已经开始成为走向通才代理的自然步骤。然而，与最近的LMM的互动揭示了当前评估基准很难捕捉到的主要局限性。事实上，任务性能（例如，VQA准确性）本身并不能提供足够的线索来理解它们的真实能力、局限性以及这些模型在多大程度上符合人类的期望。为了完善我们对这些缺陷的理解，我们偏离了当前的评估范式，并且（1）在5个不同的轴上评估了10个最近的开源LMM，从3B到80B参数尺度；幻觉、弃权、组合性、可解释性和指令遵循。我们对这些轴的评估揭示了LMMs的主要缺陷。虽然当前调整这些模型的首选解决方案是基于培训，如指令调整或RLHF，但我们宁愿（2）探索免培训情境学习（ICL）作为解决方案，并研究它如何影响这些限制。基于我们的ICL研究，（3）我们进一步推动ICL，并提出新的多模态ICL变体，如；多任务——ICL，后见之明链——ICL，和自我纠正——ICL。我们的发现如下。（1）尽管LMM取得了成功，但它们仍有缺陷，仅通过扩展无法解决。（2）ICL对LMMs缺陷的影响是微妙的；尽管ICL对提高可解释性和答案弃权很有效，但它只是稍微提高了指令遵循，并没有提高写作能力，实际上甚至放大了幻觉。（3）建议的ICL变体作为有效解决其中一些缺陷的事后方法是有希望的。代码可在以下网址获得：https://github.com/mshukor/EvALign-ICL。
摘要: Following the success of Large Language Models (LLMs), Large Multimodal Models (LMMs), such as the Flamingo model and its subsequent competitors, have started to emerge as natural steps towards generalist agents. However, interacting with recent LMMs reveals major limitations that are hardly captured by the current evaluation benchmarks. Indeed, task performances (e.g., VQA accuracy) alone do not provide enough clues to understand their real capabilities, limitations, and to which extent such models are aligned to human expectations. To refine our understanding of those flaws, we deviate from the current evaluation paradigm, and (1) evaluate 10 recent open-source LMMs from 3B up to 80B parameter scale, on 5 different axes; hallucinations, abstention, compositionality, explainability and instruction following. Our evaluation on these axes reveals major flaws in LMMs. While the current go-to solution to align these models is based on training, such as instruction tuning or RLHF, we rather (2) explore the training-free in-context learning (ICL) as a solution, and study how it affects these limitations. Based on our ICL study, (3) we push ICL further and propose new multimodal ICL variants such as; Multitask-ICL, Chain-of-Hindsight-ICL, and Self-Correcting-ICL. Our findings are as follows. (1) Despite their success, LMMs have flaws that remain unsolved with scaling alone. (2) The effect of ICL on LMMs flaws is nuanced; despite its effectiveness for improved explainability, answer abstention, ICL only slightly improves instruction following, does not improve compositional abilities, and actually even amplifies hallucinations. (3) The proposed ICL variants are promising as post-hoc approaches to efficiently tackle some of those flaws. The code is available here: https://github.com/mshukor/EvALign-ICL.
标题: Getting the Ball Rolling: Learning a Dexterous Policy for a Biomimetic Tendon-Driven Hand with Rolling Contact Joints
作者: Yasunori Toshimitsu, Benedek Forrai, Barnabas Gavin Cangan
PubTime: 2024-01-22
Downlink: http://arxiv.org/abs/2308.02453v3
Project: https://srl-ethz.github.io/get-ball-rolling/|https://youtu.be/YahsMhqNU8o|
GitHub: https://github.com/srl-ethz/faive_gym_oss|
摘要: Biomimetic, dexterous robotic hands have the potential to replicate much of the tasks that a human can do, and to achieve status as a general manipulation platform. Recent advances in reinforcement learning (RL) frameworks have achieved remarkable performance in quadrupedal locomotion and dexterous manipulation tasks. Combined with GPU-based highly parallelized simulations capable of simulating thousands of robots in parallel, RL-based controllers have become more scalable and approachable. However, in order to bring RL-trained policies to the real world, we require training frameworks that output policies that can work with physical actuators and sensors as well as a hardware platform that can be manufactured with accessible materials yet is robust enough to run interactive policies. This work introduces the biomimetic tendon-driven Faive Hand and its system architecture, which uses tendon-driven rolling contact joints to achieve a 3D printable, robust high-DoF hand design. We model each element of the hand and integrate it into a GPU simulation environment to train a policy with RL, and achieve zero-shot transfer of a dexterous in-hand sphere rotation skill to the physical robot hand.
标题: Fully Independent Communication in Multi-Agent Reinforcement Learning
作者: Rafael Pina, Varuna De Silva, Corentin Artaud
PubTime: 2024-01-26
Downlink: http://arxiv.org/abs/2401.15059v1
摘要: Multi-Agent Reinforcement Learning (MARL) comprises a broad area of research within the field of multi-agent systems. Several recent works have focused specifically on the study of communication approaches in MARL. While multiple communication methods have been proposed, these might still be too complex and not easily transferable to more practical contexts. One of the reasons for that is due to the use of the famous parameter sharing trick. In this paper, we investigate how independent learners in MARL that do not share parameters can communicate. We demonstrate that this setting might incur into some problems, to which we propose a new learning scheme as a solution. Our results show that, despite the challenges, independent agents can still learn communication strategies following our method. Additionally, we use this method to investigate how communication in MARL is affected by different network capacities, both for sharing and not sharing parameters. We observe that communication may not always be needed and that the chosen agent network sizes need to be considered when used together with communication in order to achieve efficient learning.
== Imitation Learning ==
标题: Dual RL: Unification and New Methods for Reinforcement and Imitation Learning
作者: Harshit Sikchi, Qinqing Zheng, Amy Zhang
PubTime: 2024-01-26
Downlink: http://arxiv.org/abs/2302.08560v3
Project: https://hari-sikchi.github.io/dual-rl|
中文摘要: 强化学习（RL）的目标是找到一个最大化预期累积回报的策略。已经表明，这个目标可以表示为线性约束下状态——行动访问分配的优化问题。这个公式的对偶问题，我们称之为对偶RL，是无约束的，更容易优化。在这项工作中，我们首先将几个最先进的离线RL和离线模仿学习（IL）算法作为具有共享结构的双重RL方法的实例。这种统一使我们能够确定现有方法缺点的根本原因。对于离线IL，我们的分析表明，现有的方法是基于限制性覆盖假设，这极大地限制了它们在实践中的性能。为了解决这一限制，我们提出了一种新的无鉴别器方法反冲，该方法从任意非策略数据中学习模仿，以获得接近专家的性能。对于离线RL，我们的分析将最近的离线RL方法XQL框架在对偶框架中，我们进一步提出了一种新的方法f-DVL，它为Gumbel回归损失提供了替代选择，修复了XQL的已知训练不稳定性问题。我们提出的两种方法反冲和f-DVL在IL和RL中的性能改进在大量模拟机器人运动和操纵任务中得到验证。项目代码和细节可以在这个https：//hari-sikchi.github.io/dual-rl。
摘要: The goal of reinforcement learning (RL) is to find a policy that maximizes the expected cumulative return. It has been shown that this objective can be represented as an optimization problem of state-action visitation distribution under linear constraints. The dual problem of this formulation, which we refer to as dual RL, is unconstrained and easier to optimize. In this work, we first cast several state-of-the-art offline RL and offline imitation learning (IL) algorithms as instances of dual RL approaches with shared structures. Such unification allows us to identify the root cause of the shortcomings of prior methods. For offline IL, our analysis shows that prior methods are based on a restrictive coverage assumption that greatly limits their performance in practice. To fix this limitation, we propose a new discriminator-free method ReCOIL that learns to imitate from arbitrary off-policy data to obtain near-expert performance. For offline RL, our analysis frames a recent offline RL method XQL in the dual framework, and we further propose a new method f-DVL that provides alternative choices to the Gumbel regression loss that fixes the known training instability issue of XQL. The performance improvements by both of our proposed methods, ReCOIL and f-DVL, in IL and RL are validated on an extensive suite of simulated robot locomotion and manipulation tasks. Project code and details can be found at this https://hari-sikchi.github.io/dual-rl.
标题: Multi-task robot data for dual-arm fine manipulation
作者: Heecheol Kim, Yoshiyuki Ohmura, Yasuo Kuniyoshi
PubTime: 2024-01-26
Downlink: http://arxiv.org/abs/2401.07603v2
Project: https://sites.google.com/view/multi-task-fine|https://sites.google.com/view/multi-task-fine|
中文摘要: 在机器人操纵领域，深度模仿学习被认为是一种很有前途的获得操纵技能的方法。此外，从不同的机器人数据集学习被认为是实现多功能性和适应性的可行方法。在这样的研究中，通过学习各种任务，机器人实现了跨多个对象的通用性。然而，这种多任务机器人数据集主要集中在相对不精确的单臂任务上，而没有解决机器人在现实世界中预期执行的细粒度对象操作。本文介绍了一个不同对象操作的数据集，包括双臂任务和/或需要精细操作的任务。为此，我们生成了224k集（150小时，1,104种语言指令）的数据集，其中包括双臂精细任务，如移动碗、打开铅笔盒或剥香蕉，这些数据是公开可用的。此外，该数据集包括视觉注意力信号以及双动作标签，该信号将动作分成稳健的到达轨迹和与对象的精确交互，以及实现稳健和精确的对象操作的语言指令。我们将该数据集应用于我们的双动作和注意力（DAA），这是一个为细粒度双臂操作任务设计的模型，对协变量偏移具有鲁棒性。该模型在实际机器人操作任务中进行了超过7k次试验，证明了其精细操作能力。该数据集可在https://sites.google.com/view/multi-task-fine查阅。
摘要: In the field of robotic manipulation, deep imitation learning is recognized as a promising approach for acquiring manipulation skills. Additionally, learning from diverse robot datasets is considered a viable method to achieve versatility and adaptability. In such research, by learning various tasks, robots achieved generality across multiple objects. However, such multi-task robot datasets have mainly focused on single-arm tasks that are relatively imprecise, not addressing the fine-grained object manipulation that robots are expected to perform in the real world. This paper introduces a dataset of diverse object manipulations that includes dual-arm tasks and/or tasks requiring fine manipulation. To this end, we have generated dataset with 224k episodes (150 hours, 1,104 language instructions) which includes dual-arm fine tasks such as bowl-moving, pencil-case opening or banana-peeling, and this data is publicly available. Additionally, this dataset includes visual attention signals as well as dual-action labels, a signal that separates actions into a robust reaching trajectory and precise interaction with objects, and language instructions to achieve robust and precise object manipulation. We applied the dataset to our Dual-Action and Attention (DAA), a model designed for fine-grained dual arm manipulation tasks and robust against covariate shifts. The model was tested with over 7k total trials in real robot manipulation tasks, demonstrating its capability in fine manipulation. The dataset is available at https://sites.google.com/view/multi-task-fine.
标题: Multi-Object Navigation in real environments using hybrid policies
作者: Assem Sadek, Guillaume Bono, Boris Chidlovskii
PubTime: 2024-01-24
Downlink: http://arxiv.org/abs/2401.13800v1
中文摘要: 导航已经通过SLAM和规划的结合在机器人学中得到了经典的解决。最近，除了航路点规划之外，涉及（视觉）高级推理的重要组成部分的问题已经在模拟环境中进行了探索，主要通过大规模机器学习来解决，特别是RL、离线RL或模仿学习。这些方法要求代理学习各种技能，如局部规划、映射对象和查询学习的空间表示。与航路点规划（PointGoal）等更简单的任务相比，对于这些更复杂的任务，当前最先进的模型已经在模拟中进行了彻底的评估，但据我们所知，还没有在真实环境中进行评估。在这项工作中，我们重点关注sim2real传输。我们针对具有挑战性的多对象导航（Multi-ON）任务，并将其移植到包含原始虚拟Multi-ON对象的真实副本的物理环境中。我们引入了一种混合导航方法，该方法将问题分解为两种不同的技能：（1）航路点导航用经典的SLAM结合符号规划器来处理，而（2）探索、语义映射和目标检索用结合监督学习和RL训练的深度神经网络来处理。我们在模拟和真实环境中展示了这种方法与端到端方法相比的优势，并且在这项任务中优于SOTA。
摘要: Navigation has been classically solved in robotics through the combination of SLAM and planning. More recently, beyond waypoint planning, problems involving significant components of (visual) high-level reasoning have been explored in simulated environments, mostly addressed with large-scale machine learning, in particular RL, offline-RL or imitation learning. These methods require the agent to learn various skills like local planning, mapping objects and querying the learned spatial representations. In contrast to simpler tasks like waypoint planning (PointGoal), for these more complex tasks the current state-of-the-art models have been thoroughly evaluated in simulation but, to our best knowledge, not yet in real environments. In this work we focus on sim2real transfer. We target the challenging Multi-Object Navigation (Multi-ON) task and port it to a physical environment containing real replicas of the originally virtual Multi-ON objects. We introduce a hybrid navigation method, which decomposes the problem into two different skills: (1) waypoint navigation is addressed with classical SLAM combined with a symbolic planner, whereas (2) exploration, semantic mapping and goal retrieval are dealt with deep neural networks trained with a combination of supervised learning and RL. We show the advantages of this approach compared to end-to-end methods both in simulation and a real environment and outperform the SOTA for this task.
标题: Mimicking the Maestro: Exploring the Efficacy of a Virtual AI Teacher in Fine Motor Skill Acquisition
作者: Hadar Mulian, Segev Shlomov, Lior Limonad
PubTime: 2024-01-24
Downlink: http://arxiv.org/abs/2310.10280v2
中文摘要: 运动技能，尤其是像书写这样的精细运动技能，在学术追求和日常生活中起着至关重要的作用。教授这些技能的传统方法虽然有效，但可能耗时且不一致。随着机器人技术和人工智能等先进技术的兴起，人们越来越有兴趣通过人机和人机交互，利用这些技术实现教学过程的自动化。在这项研究中，我们研究了虚拟人工智能教师在模仿人类教育者的运动技能习得技术方面的潜力。我们引入了一个人工智能教师模型，它捕捉了人类教师的独特特征。使用一个为模拟教师——学习者互动而定制的强化学习环境，我们根据四个指导性假设测试了我们的人工智能模型，强调提高了学习者的表现，提高了技能获得率，并减少了学习结果的可变性。我们的发现，在合成学习者身上得到验证，揭示了所有测试假设的显著改进。值得注意的是，我们的模型展示了跨不同学习者和环境的鲁棒性，并展示了对手写的适应性。这项研究强调了将强化学习和模仿学习模型与机器人技术相结合的潜力，从而彻底改变了关键运动技能的教学。
摘要: Motor skills, especially fine motor skills like handwriting, play an essential role in academic pursuits and everyday life. Traditional methods to teach these skills, although effective, can be time-consuming and inconsistent. With the rise of advanced technologies like robotics and artificial intelligence, there is increasing interest in automating such teaching processes using these technologies, via human-robot and human-computer interactions. In this study, we examine the potential of a virtual AI teacher in emulating the techniques of human educators for motor skill acquisition. We introduce an AI teacher model that captures the distinct characteristics of human instructors. Using a Reinforcement Learning environment tailored to mimic teacher-learner interactions, we tested our AI model against four guiding hypotheses, emphasizing improved learner performance, enhanced rate of skill acquisition, and reduced variability in learning outcomes. Our findings, validated on synthetic learners, revealed significant improvements across all tested hypotheses. Notably, our model showcased robustness across different learners and settings and demonstrated adaptability to handwriting. This research underscores the potential of integrating Reinforcement Learning and Imitation Learning models with robotics in revolutionizing the teaching of critical motor skills.
标题: Adversarial Imitation Learning from Visual Observations using Latent Information
作者: Vittorio Giammarino, James Queeney, Ioannis Ch. Paschalidis
PubTime: 2024-01-23
Downlink: http://arxiv.org/abs/2309.17371v2
中文摘要: 我们专注于从视觉观察中模仿学习的问题，其中学习代理可以访问专家的视频作为其唯一的学习来源。该框架的挑战包括缺乏专家行动和环境的部分可观测性，因为地面真实状态只能从像素中推断出来。为了解决这个问题，我们首先对部分可观察环境中的模仿学习进行了理论分析。我们建立了关于专家和代理潜在状态转移分布之间的散度的学习代理的次优性的上界。在这种分析的激励下，我们引入了一种称为来自观察的潜在对抗性模仿的算法，该算法将非策略对抗性模仿技术与来自观察序列的代理状态的学习潜在表示相结合。在高维连续机器人任务的实验中，我们表明我们的算法符合最先进的性能，同时提供了显著的计算优势。此外，我们展示了我们的方法如何通过利用专家视频来提高像素强化学习的效率。为了确保可重复性，我们提供了对我们代码的免费访问。
摘要: We focus on the problem of imitation learning from visual observations, where the learning agent has access to videos of experts as its sole learning source. The challenges of this framework include the absence of expert actions and the partial observability of the environment, as the ground-truth states can only be inferred from pixels. To tackle this problem, we first conduct a theoretical analysis of imitation learning in partially observable environments. We establish upper bounds on the suboptimality of the learning agent with respect to the divergence between the expert and the agent latent state-transition distributions. Motivated by this analysis, we introduce an algorithm called Latent Adversarial Imitation from Observations, which combines off-policy adversarial imitation techniques with a learned latent representation of the agent's state from sequences of observations. In experiments on high-dimensional continuous robotic tasks, we show that our algorithm matches state-of-the-art performance while providing significant computational advantages. Additionally, we show how our method can be used to improve the efficiency of reinforcement learning from pixels by leveraging expert videos. To ensure reproducibility, we provide free access to our code.
标题: Sample-efficient Adversarial Imitation Learning
作者: Dahuin Jung, Hyungyu Lee, Sungroh Yoon
PubTime: 2024-01-23
Downlink: http://arxiv.org/abs/2303.07846v2
摘要: Imitation learning, in which learning is performed by demonstration, has been studied and advanced for sequential decision-making tasks in which a reward function is not predefined. However, imitation learning methods still require numerous expert demonstration samples to successfully imitate an expert's behavior. To improve sample efficiency, we utilize self-supervised representation learning, which can generate vast training signals from the given data. In this study, we propose a self-supervised representation-based adversarial imitation learning method to learn state and action representations that are robust to diverse distortions and temporally predictive, on non-image control tasks. In particular, in comparison with existing self-supervised learning methods for tabular data, we propose a different corruption method for state and action representations that is robust to diverse distortions. We theoretically and empirically observe that making an informative feature manifold with less sample complexity significantly improves the performance of imitation learning. The proposed method shows a 39% relative improvement over existing adversarial imitation learning methods on MuJoCo in a setting limited to 100 expert state-action pairs. Moreover, we conduct comprehensive ablations and additional experiments using demonstrations with varying optimality to provide insights into a range of factors.
== robotic agent ==
标题: Generative Expressive Robot Behaviors using Large Language Models
作者: Karthik Mahadevan, Jonathan Chien, Noah Brown
PubTime: 2024-01-26
Downlink: http://arxiv.org/abs/2401.14673v1
Project: https://generative-expressive-motion.github.io/|
中文摘要: 人们使用表达行为来有效地与他人交流和协调他们的行动，例如点头表示对瞥他们一眼的人的认可，或者在繁忙的走廊上说“对不起”从人群中经过。我们希望机器人也能在人机交互中表现出富有表现力的行为。先前的工作提出了基于规则的方法，这些方法很难扩展到新的通信模式或社交场合，而数据驱动的方法需要针对机器人所处的每个社交场合的专门数据集。我们建议利用大型语言模型（LLMs）提供的丰富的社会背景及其基于指令或用户偏好生成运动的能力，来生成具有适应性和可组合性的机器人运动，并相互构建。我们的方法利用机器人可用和学习的技能，利用少量思维链提示将人类语言指令翻译成参数化的控制代码。通过用户研究和模拟实验，我们证明了我们的方法产生了用户认为有能力和容易理解的行为。补充材料可以在https：//generative-expressive-motion.github.io/。
摘要: People employ expressive behaviors to effectively communicate and coordinate their actions with others, such as nodding to acknowledge a person glancing at them or saying "excuse me" to pass people in a busy corridor. We would like robots to also demonstrate expressive behaviors in human-robot interaction. Prior work proposes rule-based methods that struggle to scale to new communication modalities or social situations, while data-driven methods require specialized datasets for each social situation the robot is used in. We propose to leverage the rich social context available from large language models (LLMs) and their ability to generate motion based on instructions or user preferences, to generate expressive robot motion that is adaptable and composable, building upon each other. Our approach utilizes few-shot chain-of-thought prompting to translate human language instructions into parametrized control code using the robot's available and learned skills. Through user studies and simulation experiments, we demonstrate that our approach produces behaviors that users found to be competent and easy to understand. Supplementary material can be found at https://generative-expressive-motion.github.io/.
标题: MResT: Multi-Resolution Sensing for Real-Time Control with Vision-Language Models
作者: Saumya Saxena, Mohit Sharma, Oliver Kroemer
PubTime: 2024-01-25
Downlink: http://arxiv.org/abs/2401.14502v1
Project: http://tinyurl.com/multi-res-realtime-control|
摘要: Leveraging sensing modalities across diverse spatial and temporal resolutions can improve performance of robotic manipulation tasks. Multi-spatial resolution sensing provides hierarchical information captured at different spatial scales and enables both coarse and precise motions. Simultaneously multi-temporal resolution sensing enables the agent to exhibit high reactivity and real-time control. In this work, we propose a framework, MResT (Multi-Resolution Transformer), for learning generalizable language-conditioned multi-task policies that utilize sensing at different spatial and temporal resolutions using networks of varying capacities to effectively perform real time control of precise and reactive tasks. We leverage off-the-shelf pretrained vision-language models to operate on low-frequency global features along with small non-pretrained models to adapt to high frequency local feedback. Through extensive experiments in 3 domains (coarse, precise and dynamic manipulation tasks), we show that our approach significantly improves (2X on average) over recent multi-task baselines. Further, our approach generalizes well to visual and geometric variations in target objects and to varying interaction forces.
标题: The Conversation is the Command: Interacting with Real-World Autonomous Robot Through Natural Language
作者: Linus Nwankwo, Elmar Rueckert
PubTime: 2024-01-22
Downlink: http://arxiv.org/abs/2401.11838v1
Project: https://osf.io/wzyf6|
GitHub: https://github.com/LinusNEP/TCC_IRoNL.git)|
中文摘要: 近年来，自主代理在现实世界环境中激增，如我们的家庭、办公室和公共场所。然而，自然的人机交互仍然是一个关键的挑战。在本文中，我们介绍了一种协同利用大型语言模型（LLMs）和多模态视觉语言模型（VLMs）的能力的方法，使人类能够通过对话与自主机器人进行自然交互。我们利用LLMs解码来自人类的高级自然语言指令，并将其抽象为精确的机器人可操作命令或查询。此外，我们利用VLMs来提供对机器人任务环境的视觉和语义理解。我们99.13%的命令识别准确率和97.96%的命令执行成功率表明，我们的方法可以增强现实世界应用中的人机交互。本文的视频演示可以在https://osf.io/wzyf6找到，代码可以在我们的GitHub资源库（https://github.com/LinusNEP/tcc_iron.git）找到
摘要: In recent years, autonomous agents have surged in real-world environments such as our homes, offices, and public spaces. However, natural human-robot interaction remains a key challenge. In this paper, we introduce an approach that synergistically exploits the capabilities of large language models (LLMs) and multimodal vision-language models (VLMs) to enable humans to interact naturally with autonomous robots through conversational dialogue. We leveraged the LLMs to decode the high-level natural language instructions from humans and abstract them into precise robot actionable commands or queries. Further, we utilised the VLMs to provide a visual and semantic understanding of the robot's task environment. Our results with 99.13% command recognition accuracy and 97.96% commands execution success show that our approach can enhance human-robot interaction in real-world applications. The video demonstrations of this paper can be found at https://osf.io/wzyf6 and the code is available at our GitHub repository (https://github.com/LinusNEP/TCC_IRoNL.git).
标题: Security Considerations in AI-Robotics: A Survey of Current Methods, Challenges, and Opportunities
作者: Subash Neupane, Shaswata Mitra, Ivan A. Fernandez
PubTime: 2024-01-26
Downlink: http://arxiv.org/abs/2310.08565v3
中文摘要: 机器人和人工智能(AI)从一开始就密不可分。今天，人工智能机器人系统已经成为我们日常生活中不可或缺的一部分，从机器人吸尘器到半自动汽车。这些系统建立在三个基本架构元素之上：感知、导航和规划以及控制。然而，尽管人工智能机器人系统的集成提高了我们的生活质量，但它也带来了一个严重的问题——这些系统容易受到安全攻击。构成人工智能机器人系统的物理组件、算法和数据可能会被恶意行为者利用，潜在地导致可怕的后果。出于解决人工智能机器人系统中安全问题的需要，本文提出了一个跨三个维度的全面调查和分类：攻击面，伦理和法律问题，以及人机交互（HRI）安全。我们的目标是为用户、开发者和其他利益相关者提供对这些领域的整体理解，以增强整体人工智能机器人系统的安全性。我们从调查潜在的攻击面开始，并提供缓解防御策略。然后，我们深入研究伦理问题，如依赖性和心理影响，以及关于这些系统问责制的法律问题。此外，还讨论了HRI等新兴趋势，考虑了隐私、完整性、安全性、可信度和可解释性问题。最后，我们提出了我们对这一动态和有前途的领域的未来研究方向的愿景。
摘要: Robotics and Artificial Intelligence (AI) have been inextricably intertwined since their inception. Today, AI-Robotics systems have become an integral part of our daily lives, from robotic vacuum cleaners to semi-autonomous cars. These systems are built upon three fundamental architectural elements: perception, navigation and planning, and control. However, while the integration of AI-Robotics systems has enhanced the quality our lives, it has also presented a serious problem - these systems are vulnerable to security attacks. The physical components, algorithms, and data that make up AI-Robotics systems can be exploited by malicious actors, potentially leading to dire consequences. Motivated by the need to address the security concerns in AI-Robotics systems, this paper presents a comprehensive survey and taxonomy across three dimensions: attack surfaces, ethical and legal concerns, and Human-Robot Interaction (HRI) security. Our goal is to provide users, developers and other stakeholders with a holistic understanding of these areas to enhance the overall AI-Robotics system security. We begin by surveying potential attack surfaces and provide mitigating defensive strategies. We then delve into ethical issues, such as dependency and psychological impact, as well as the legal concerns regarding accountability for these systems. Besides, emerging trends such as HRI are discussed, considering privacy, integrity, safety, trustworthiness, and explainability concerns. Finally, we present our vision for future research directions in this dynamic and promising field.
标题: GCBF+: A Neural Graph Control Barrier Function Framework for Distributed Safe Multi-Agent Control
作者: Songyuan Zhang, Oswin So, Kunal Garg
PubTime: 2024-01-25
Downlink: http://arxiv.org/abs/2401.14554v1
中文摘要: 大规模多智能体系统（MAS）的分布式、可扩展和安全控制是一个具有挑战性的问题。在本文中，我们设计了一个分布式框架，用于在有障碍的大规模环境中进行安全的多智能体控制，在这种环境中，需要大量的智能体仅使用本地信息来维护安全并到达它们的目标位置。我们引入了一类新的证书，称为图控制屏障函数（GCBF），它基于成熟的安全保证控制屏障函数（CBF）理论，并利用图结构实现MAS的可扩展和可推广的分布式控制。我们发展了一个新的理论框架来证明具有单个GCBF的任意大小的MAS的安全性。我们提出了一个新的训练框架GCBF+，它使用图形神经网络（GNNs）来参数化候选GCBF和分布式控制策略。所提出的框架是分布式的，并且能够直接从激光雷达获取点云，而不是实际的状态信息，用于真实世界的机器人应用。我们通过在一群无人机上进行的各种硬件实验来说明所提出的方法的有效性，这些无人机的目标范围从交换位置到停靠在移动目标上而不发生碰撞。此外，我们进行了大量的数值实验，其中代理的数量和密度，以及障碍的数量，增加。经验结果表明，在具有非线性智能体的复杂环境中（例如，Crazyflie无人机），GCBF+优于手工制作的基于CBF的方法，对于多达256个智能体的相对小规模MAS，其最佳性能高达20%，对于具有1024个智能体的MAS，领先强化学习（RL）方法高达40%。此外，所提出的方法在实现高安全率方面不会损害性能，这是基于RL的方法中常见的权衡。
摘要: Distributed, scalable, and safe control of large-scale multi-agent systems (MAS) is a challenging problem. In this paper, we design a distributed framework for safe multi-agent control in large-scale environments with obstacles, where a large number of agents are required to maintain safety using only local information and reach their goal locations. We introduce a new class of certificates, termed graph control barrier function (GCBF), which are based on the well-established control barrier function (CBF) theory for safety guarantees and utilize a graph structure for scalable and generalizable distributed control of MAS. We develop a novel theoretical framework to prove the safety of an arbitrary-sized MAS with a single GCBF. We propose a new training framework GCBF+ that uses graph neural networks (GNNs) to parameterize a candidate GCBF and a distributed control policy. The proposed framework is distributed and is capable of directly taking point clouds from LiDAR, instead of actual state information, for real-world robotic applications. We illustrate the efficacy of the proposed method through various hardware experiments on a swarm of drones with objectives ranging from exchanging positions to docking on a moving target without collision. Additionally, we perform extensive numerical experiments, where the number and density of agents, as well as the number of obstacles, increase. Empirical results show that in complex environments with nonlinear agents (e.g., Crazyflie drones) GCBF+ outperforms the handcrafted CBF-based method with the best performance by up to 20% for relatively small-scale MAS for up to 256 agents, and leading reinforcement learning (RL) methods by up to 40% for MAS with 1024 agents. Furthermore, the proposed method does not compromise on the performance, in terms of goal reaching, for achieving high safety rates, which is a common trade-off in RL-based methods.
标题: Toward Family-Robot Interactions: A Family-Centered Framework in HRI
作者: Bengisu Cagiltay, Bilge Mutlu
PubTime: 2024-01-25
Downlink: http://arxiv.org/abs/2401.14478v1
摘要: As robotic products become more integrated into daily life, there is a greater need to understand authentic and real-world human-robot interactions to inform product design. Across many domestic, educational, and public settings, robots interact with not only individuals and groups of users, but also families, including children, parents, relatives, and even pets. However, products developed to date and research in human-robot and child-robot interactions have focused on the interaction with their primary users, neglecting the complex and multifaceted interactions between family members and with the robot. There is a significant gap in knowledge, methods, and theories for how to design robots to support these interactions. To inform the design of robots that can support and enhance family life, this paper provides (1) a narrative review exemplifying the research gap and opportunities for family-robot interactions and (2) an actionable family-centered framework for research and practices in human-robot and child-robot interaction.
== Object Detection ==
标题: MGTUNet: An new UNet for colon nuclei instance segmentation and quantification
作者: Liangrui Pan, Lian Wang, Zhichao Feng
PubTime: 2024-01-26
Downlink: http://arxiv.org/abs/2210.10981v2
Project: https://doi.org/10.1109/BIBM55620.2022.9995669|
中文摘要: 结直肠癌（CRC）是发病率和死亡率最高的三种恶性肿瘤类型之一。组织病理学图像是诊断结肠癌的金标准。细胞核实例分割和分类以及核成分回归任务可以帮助分析结肠组织中的肿瘤微环境。传统方法仍然无法同时处理这两种类型的任务，并且预测精度差，应用成本高。本文提出了一种新的基于UNet框架的核处理UNet模型，称为MGTUNet，它使用Mish、组归一化和转置卷积层来改进分割模型，并使用ranger优化器来调整SmoothL1Loss值。其次，使用不同的通道对不同类型的核进行分割和分类，最终同时完成核实例分割和分类任务以及核成分回归任务。最后，我们使用八个分割模型进行了广泛的比较实验。通过比较三个评估指标和模型的参数大小，MGTUNet在PQ上获得0.6254，在mPQ上获得0.6359，在R2上获得0.8695。因此，实验证明MGTUNet现在是量化结肠癌组织病理学图像的最先进方法。
摘要: Colorectal cancer (CRC) is among the top three malignant tumor types in terms of morbidity and mortality. Histopathological images are the gold standard for diagnosing colon cancer. Cellular nuclei instance segmentation and classification, and nuclear component regression tasks can aid in the analysis of the tumor microenvironment in colon tissue. Traditional methods are still unable to handle both types of tasks end-to-end at the same time, and have poor prediction accuracy and high application costs. This paper proposes a new UNet model for handling nuclei based on the UNet framework, called MGTUNet, which uses Mish, Group normalization and transposed convolution layer to improve the segmentation model, and a ranger optimizer to adjust the SmoothL1Loss values. Secondly, it uses different channels to segment and classify different types of nucleus, ultimately completing the nuclei instance segmentation and classification task, and the nuclei component regression task simultaneously. Finally, we did extensive comparison experiments using eight segmentation models. By comparing the three evaluation metrics and the parameter sizes of the models, MGTUNet obtained 0.6254 on PQ, 0.6359 on mPQ, and 0.8695 on R2. Thus, the experiments demonstrated that MGTUNet is now a state-of-the-art method for quantifying histopathological images of colon cancer.
标题: The Endoscapes Dataset for Surgical Scene Segmentation, Object Detection, and Critical View of Safety Assessment: Official Splits and Benchmark
作者: Aditya Murali, Deepak Alapatt, Pietro Mascagni
PubTime: 2024-01-26
Downlink: http://arxiv.org/abs/2312.12429v2
GitHub: https://github.com/CAMMA-public/Endoscapes|
中文摘要: 本技术报告提供了Endoscapes的详细概述，Endoscapes是一个腹腔镜胆囊切除术（LC）视频数据集，带有高度复杂的注释，旨在自动评估安全性关键观点（CVS）。Endoscapes包括201个LC视频，其帧被稀疏但有规律地注释，带有分割掩码、边界框和由三个不同的临床专家进行的CVS评估。总共有来自201个视频的11090个用CVS注释的帧和1933个用工具和解剖边界框注释的帧，以及来自201个视频中的50个用工具和解剖分割遮罩注释的另外422个帧。在这份报告中，我们提供了详细的数据集统计数据（大小、类别分布、数据集分割等。）和全面的性能基准，用于实例分割、对象检测和CVS预测。数据集和模型检查点可在https：//github.com/CAMMA-public/Endoscapes公开获得。
摘要: This technical report provides a detailed overview of Endoscapes, a dataset of laparoscopic cholecystectomy (LC) videos with highly intricate annotations targeted at automated assessment of the Critical View of Safety (CVS). Endoscapes comprises 201 LC videos with frames annotated sparsely but regularly with segmentation masks, bounding boxes, and CVS assessment by three different clinical experts. Altogether, there are 11090 frames annotated with CVS and 1933 frames annotated with tool and anatomy bounding boxes from the 201 videos, as well as an additional 422 frames from 50 of the 201 videos annotated with tool and anatomy segmentation masks. In this report, we provide detailed dataset statistics (size, class distribution, dataset splits, etc.) and a comprehensive performance benchmark for instance segmentation, object detection, and CVS prediction. The dataset and model checkpoints are publically available at https://github.com/CAMMA-public/Endoscapes.
标题: MagicDrive: Street View Generation with Diverse 3D Geometry Control
作者: Ruiyuan Gao, Kai Chen, Enze Xie
PubTime: 2024-01-26
Downlink: http://arxiv.org/abs/2310.02601v5
Project: https://flymin.github.io/magicdrive|
中文摘要: 扩散模型的最新进展显著增强了2D控制的数据合成。然而，对3D感知任务至关重要的街景生成中的精确3D控制仍然难以捉摸。具体来说，利用鸟瞰（BEV）作为主要条件通常会导致几何控制（例如，高度）方面的挑战，影响对象形状、遮挡模式和路面高程的表示，所有这些对于感知数据合成都是必不可少的，尤其是对于3D对象检测任务。在本文中，我们介绍了MagicDrive，这是一个新颖的街景生成框架，提供了各种3D几何控制，包括相机姿势、道路地图和3D边界框，以及文本描述，通过定制的编码策略实现。此外，我们的设计包含一个交叉视图注意模块，确保多个摄像头视图的一致性。通过MagicDrive，我们实现了高保真街景合成，捕捉了细微的3D几何图形和各种场景描述，增强了BEV分割和3D对象检测等任务。
摘要: Recent advancements in diffusion models have significantly enhanced the data synthesis with 2D control. Yet, precise 3D control in street view generation, crucial for 3D perception tasks, remains elusive. Specifically, utilizing Bird's-Eye View (BEV) as the primary condition often leads to challenges in geometry control (e.g., height), affecting the representation of object shapes, occlusion patterns, and road surface elevations, all of which are essential to perception data synthesis, especially for 3D object detection tasks. In this paper, we introduce MagicDrive, a novel street view generation framework offering diverse 3D geometry controls, including camera poses, road maps, and 3D bounding boxes, together with textual descriptions, achieved through tailored encoding strategies. Besides, our design incorporates a cross-view attention module, ensuring consistency across multiple camera views. With MagicDrive, we achieve high-fidelity street-view synthesis that captures nuanced 3D geometry and various scene descriptions, enhancing tasks like BEV segmentation and 3D object detection.
标题: Expectation Maximization Pseudo Labels
作者: Moucheng Xu, Yukun Zhou, Chen Jin
PubTime: 2024-01-26
Downlink: http://arxiv.org/abs/2305.01747v2
GitHub: https://github.com/moucheng2017/EMSSL|
中文摘要: 在本文中，我们研究伪标记。伪标记采用对未标记数据的原始推理作为自我训练的伪标签。我们通过在这种技术和期望最大化算法之间建立联系来阐明伪标记的经验成功。通过这一点，我们意识到最初的伪标签是对其更全面的基础公式的经验估计。根据这一认识，我们提出了贝叶斯定理下伪标签的完全推广，称为贝叶斯伪标签。随后，我们引入了一种变分方法来生成这些贝叶斯伪标签，包括学习阈值以自动选择高质量的伪标签。在本文的其余部分，我们展示了伪标记及其广义形式贝叶斯伪标记在医学图像半监督分割中的应用。具体来说，我们关注：1）来自CT体积的肺血管的3D二进制分割；2）来自MRI体积的脑肿瘤的2D多类分割；3）来自MRI体积的全脑肿瘤的3D二进制分割；和4）来自MRI体积的前列腺的3D二进制分割。我们进一步证明了伪标签可以增强学习表示的鲁棒性。该代码发布在以下GitHub存储库中：https：//github.com/moucheng 2017/EMSSL
摘要: In this paper, we study pseudo-labelling. Pseudo-labelling employs raw inferences on unlabelled data as pseudo-labels for self-training. We elucidate the empirical successes of pseudo-labelling by establishing a link between this technique and the Expectation Maximisation algorithm. Through this, we realise that the original pseudo-labelling serves as an empirical estimation of its more comprehensive underlying formulation. Following this insight, we present a full generalisation of pseudo-labels under Bayes' theorem, termed Bayesian Pseudo Labels. Subsequently, we introduce a variational approach to generate these Bayesian Pseudo Labels, involving the learning of a threshold to automatically select high-quality pseudo labels. In the remainder of the paper, we showcase the applications of pseudo-labelling and its generalised form, Bayesian Pseudo-Labelling, in the semi-supervised segmentation of medical images. Specifically, we focus on: 1) 3D binary segmentation of lung vessels from CT volumes; 2) 2D multi-class segmentation of brain tumours from MRI volumes; 3) 3D binary segmentation of whole brain tumours from MRI volumes; and 4) 3D binary segmentation of prostate from MRI volumes. We further demonstrate that pseudo-labels can enhance the robustness of the learned representations. The code is released in the following GitHub repository: https://github.com/moucheng2017/EMSSL
标题: pLitterStreet: Street Level Plastic Litter Detection and Mapping
作者: Sriram Reddy Mandhati, N. Lakmal Deshapriya, Chatura Lavanga Mendis
PubTime: 2024-01-26
Downlink: http://arxiv.org/abs/2401.14719v1
GitHub: https://github.com/gicait/pLitter|
中文摘要: 塑料污染是一个关键的环境问题，检测和监控塑料垃圾对于减轻其影响至关重要。本文介绍了绘制街道垃圾地图的方法，主要关注塑料垃圾和垃圾桶的位置。我们的方法包括采用深度学习技术来识别垃圾和废物。由安装在车辆上的摄像机拍摄的街道图像中的INS。随后，我们利用热图直观地表示l的分布遍布城市的垃圾和垃圾桶。此外，我们还提供了在我们的方法中开发和使用的开源数据集（“pLitterStreet”）的创建细节。该数据集包含从车载摄像机收集的13,000多幅完全注释的图像，并包括边界框标签。为了评估我们数据集的有效性，我们测试了四种众所周知的最先进的对象检测算法（更快的R-CNN、RetinaNet、YOLOv3和YOLOv5），实现了40%以上的平均精度（AP）。虽然结果显示了平均指标，但我们的实验证明了使用车载摄像机进行塑料垃圾测绘的可靠性。“pLitterStreet”也可以成为研究人员和从业人员开发和进一步改进现有机器学习模型的宝贵资源，用于检测和绘制城市环境中的塑料垃圾。该数据集是开放源码的，有关数据集和训练模型的更多详细信息，请访问https://github.com/gicait/pLitter。
摘要: Plastic pollution is a critical environmental issue, and detecting and monitoring plastic litter is crucial to mitigate its impact. This paper presents the methodology of mapping street-level litter, focusing primarily on plastic waste and the location of trash bins. Our methodology involves employing a deep learning technique to identify litter and trash bins from street-level imagery taken by a camera mounted on a vehicle. Subsequently, we utilized heat maps to visually represent the distribution of litter and trash bins throughout cities. Additionally, we provide details about the creation of an open-source dataset ("pLitterStreet") which was developed and utilized in our approach. The dataset contains more than 13,000 fully annotated images collected from vehicle-mounted cameras and includes bounding box labels. To evaluate the effectiveness of our dataset, we tested four well known state-of-the-art object detection algorithms (Faster R-CNN, RetinaNet, YOLOv3, and YOLOv5), achieving an average precision (AP) above 40%. While the results show average metrics, our experiments demonstrated the reliability of using vehicle-mounted cameras for plastic litter mapping. The "pLitterStreet" can also be a valuable resource for researchers and practitioners to develop and further improve existing machine learning models for detecting and mapping plastic litter in an urban environment. The dataset is open-source and more details about the dataset and trained models can be found at https://github.com/gicait/pLitter.
标题: MATIS: Masked-Attention Transformers for Surgical Instrument Segmentation
作者: Nicolás Ayobi, Alejandra Pérez-Rondón, Santiago Rodríguez
PubTime: 2024-01-26
Downlink: http://arxiv.org/abs/2303.09514v4
GitHub: https://github.com/BCV-Uniandes/MATIS|
摘要: We propose Masked-Attention Transformers for Surgical Instrument Segmentation (MATIS), a two-stage, fully transformer-based method that leverages modern pixel-wise attention mechanisms for instrument segmentation. MATIS exploits the instance-level nature of the task by employing a masked attention module that generates and classifies a set of fine instrument region proposals. Our method incorporates long-term video-level information through video transformers to improve temporal consistency and enhance mask classification. We validate our approach in the two standard public benchmarks, Endovis 2017 and Endovis 2018. Our experiments demonstrate that MATIS' per-frame baseline outperforms previous state-of-the-art methods and that including our temporal consistency module boosts our model's performance further.
专属领域论文订阅
“ VX关注晓理紫，每日更新论文，如感兴趣，请转发给有需要的同学，谢谢支持。谢谢提供建议
“ 如果你感觉对你有所帮助，请关注我，每日准时为你推送最新论文
“ 为了答谢各位网友的支持，从今日起免费为300名读者提供订阅主题论文服务，只需关注公号并在留言中提供{邮箱+论文主题}（如：123456@xx.com + chatgpt@large language model @LLM）,主题必须是同一个领域，最多三个关键词。解释权归博主所有
